Oct 29 21:45:37.216327 ip-10-0-129-146 systemd[1]: Starting Kubernetes Kubelet...
Oct 29 21:45:37.626846 ip-10-0-129-146 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 21:45:37.626846 ip-10-0-129-146 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 29 21:45:37.626846 ip-10-0-129-146 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 21:45:37.626846 ip-10-0-129-146 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 21:45:37.631465 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631392    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:45:37.631465 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631464    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:45:37.631465 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631468    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:45:37.631557 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631472    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:45:37.631557 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631475    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:45:37.631557 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631477    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:45:37.631557 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631481    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:45:37.631557 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631484    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:45:37.631557 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631487    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:45:37.631557 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631490    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:45:37.631557 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631493    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:45:37.631557 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631495    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:45:37.631557 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631498    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:45:37.631557 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631501    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:45:37.631557 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631503    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:45:37.631557 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631506    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:45:37.631557 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631509    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:45:37.631557 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631511    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:45:37.631557 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631514    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:45:37.631557 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631516    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:45:37.631557 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631519    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:45:37.631557 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631522    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:45:37.631557 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631524    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:45:37.631557 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631527    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:45:37.632048 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631529    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:45:37.632048 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631532    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:45:37.632048 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631538    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:45:37.632048 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631541    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:45:37.632048 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631544    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:45:37.632048 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631547    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:45:37.632048 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631549    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:45:37.632048 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631551    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:45:37.632048 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631554    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:45:37.632048 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631557    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:45:37.632048 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631559    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:45:37.632048 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631562    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:45:37.632048 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631565    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:45:37.632048 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631568    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:45:37.632048 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631571    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:45:37.632048 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631574    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:45:37.632048 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631576    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:45:37.632048 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631579    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:45:37.632048 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631581    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:45:37.632048 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631584    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:45:37.632519 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631586    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:45:37.632519 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631589    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:45:37.632519 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631592    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:45:37.632519 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631594    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:45:37.632519 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631597    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:45:37.632519 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631599    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:45:37.632519 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631601    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:45:37.632519 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631604    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:45:37.632519 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631607    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:45:37.632519 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631610    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:45:37.632519 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631632    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:45:37.632519 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631636    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:45:37.632519 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631639    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:45:37.632519 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631643    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:45:37.632519 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631646    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:45:37.632519 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631649    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:45:37.632519 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631652    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:45:37.632519 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631655    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:45:37.632519 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631658    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:45:37.632960 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631660    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:45:37.632960 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631663    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:45:37.632960 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631666    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:45:37.632960 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631668    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:45:37.632960 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631671    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:45:37.632960 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631674    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:45:37.632960 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631676    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:45:37.632960 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631679    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:45:37.632960 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631681    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:45:37.632960 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631684    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:45:37.632960 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631687    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:45:37.632960 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631690    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:45:37.632960 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631695    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:45:37.632960 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631698    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:45:37.632960 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631701    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:45:37.632960 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631706    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:45:37.632960 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631710    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:45:37.632960 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631714    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:45:37.632960 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631717    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:45:37.633456 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631720    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:45:37.633456 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631723    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:45:37.633456 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631727    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:45:37.633456 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631730    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:45:37.633456 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631733    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:45:37.633456 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631736    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:45:37.633456 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631739    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:45:37.633456 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631743    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:45:37.633456 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631745    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:45:37.633456 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631748    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:45:37.633456 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631751    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:45:37.633456 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631753    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:45:37.633456 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631756    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:45:37.633456 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631759    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:45:37.633456 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631762    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:45:37.633456 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631764    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:45:37.633456 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631767    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:45:37.633456 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631769    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:45:37.633456 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631772    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:45:37.633903 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631774    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:45:37.633903 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631778    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:45:37.633903 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631781    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:45:37.633903 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631783    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:45:37.633903 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.631786    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:45:37.633903 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632540    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:45:37.633903 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632546    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:45:37.633903 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632549    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:45:37.633903 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632554    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:45:37.633903 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632558    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:45:37.633903 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632561    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:45:37.633903 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632564    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:45:37.633903 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632567    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:45:37.633903 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632571    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:45:37.633903 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632574    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:45:37.633903 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632576    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:45:37.633903 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632579    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:45:37.633903 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632582    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:45:37.633903 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632585    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:45:37.634370 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632588    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:45:37.634370 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632591    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:45:37.634370 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632593    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:45:37.634370 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632596    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:45:37.634370 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632599    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:45:37.634370 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632601    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:45:37.634370 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632604    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:45:37.634370 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632607    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:45:37.634370 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632610    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:45:37.634370 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632612    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:45:37.634370 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632615    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:45:37.634370 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632617    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:45:37.634370 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632620    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:45:37.634370 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632622    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:45:37.634370 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632625    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:45:37.634370 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632627    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:45:37.634370 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632630    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:45:37.634370 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632633    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:45:37.634370 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632636    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:45:37.634370 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632638    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:45:37.634850 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632641    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:45:37.634850 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632643    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:45:37.634850 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632646    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:45:37.634850 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632649    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:45:37.634850 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632652    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:45:37.634850 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632654    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:45:37.634850 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632656    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:45:37.634850 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632660    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:45:37.634850 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632662    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:45:37.634850 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632665    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:45:37.634850 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632668    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:45:37.634850 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632670    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:45:37.634850 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632673    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:45:37.634850 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632675    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:45:37.634850 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632678    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:45:37.634850 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632680    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:45:37.634850 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632683    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:45:37.634850 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632685    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:45:37.634850 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632688    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:45:37.635306 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632691    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:45:37.635306 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632694    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:45:37.635306 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632696    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:45:37.635306 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632699    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:45:37.635306 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632701    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:45:37.635306 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632704    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:45:37.635306 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632706    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:45:37.635306 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632709    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:45:37.635306 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632711    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:45:37.635306 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632714    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:45:37.635306 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632717    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:45:37.635306 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632719    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:45:37.635306 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632722    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:45:37.635306 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632724    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:45:37.635306 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632727    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:45:37.635306 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632729    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:45:37.635306 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632732    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:45:37.635306 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632735    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:45:37.635306 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632738    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:45:37.635306 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632740    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:45:37.635774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632743    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:45:37.635774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632746    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:45:37.635774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632752    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:45:37.635774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632755    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:45:37.635774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632757    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:45:37.635774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632760    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:45:37.635774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632762    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:45:37.635774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632765    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:45:37.635774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632768    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:45:37.635774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632771    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:45:37.635774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632774    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:45:37.635774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632776    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:45:37.635774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632779    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:45:37.635774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632782    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:45:37.635774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632785    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:45:37.635774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632787    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:45:37.635774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632790    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:45:37.635774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632792    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:45:37.635774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632795    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:45:37.636232 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632799    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:45:37.636232 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632803    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:45:37.636232 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632806    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:45:37.636232 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632809    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:45:37.636232 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632811    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:45:37.636232 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632814    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:45:37.636232 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632817    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:45:37.636232 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632819    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:45:37.636232 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632822    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:45:37.636232 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632824    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:45:37.636232 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632827    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:45:37.636232 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632831    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:45:37.636232 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632834    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:45:37.636232 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.632836    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:45:37.636232 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633323    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:45:37.636232 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633331    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:45:37.636232 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633334    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:45:37.636232 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633338    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:45:37.636232 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633341    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:45:37.636688 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633344    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:45:37.636688 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633346    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:45:37.636688 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633349    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:45:37.636688 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633352    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:45:37.636688 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633355    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:45:37.636688 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633357    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:45:37.636688 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633362    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:45:37.636688 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633364    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:45:37.636688 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633367    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:45:37.636688 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633370    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:45:37.636688 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633372    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:45:37.636688 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633375    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:45:37.636688 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633377    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:45:37.636688 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633380    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:45:37.636688 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633383    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:45:37.636688 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633386    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:45:37.636688 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633388    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:45:37.636688 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633391    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:45:37.636688 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633394    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:45:37.637156 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633396    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:45:37.637156 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633399    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:45:37.637156 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633402    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:45:37.637156 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633404    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:45:37.637156 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633408    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:45:37.637156 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633410    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:45:37.637156 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633413    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:45:37.637156 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633416    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:45:37.637156 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633419    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:45:37.637156 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633422    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:45:37.637156 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633425    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:45:37.637156 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633428    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:45:37.637156 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633430    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:45:37.637156 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633433    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:45:37.637156 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633436    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:45:37.637156 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633438    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:45:37.637156 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633441    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:45:37.637156 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633444    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:45:37.637156 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633446    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:45:37.637645 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633452    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:45:37.637645 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633455    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:45:37.637645 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633459    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:45:37.637645 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633462    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:45:37.637645 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633465    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:45:37.637645 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633468    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:45:37.637645 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633471    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:45:37.637645 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633473    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:45:37.637645 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633476    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:45:37.637645 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633479    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:45:37.637645 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633481    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:45:37.637645 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633484    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:45:37.637645 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633487    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:45:37.637645 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633489    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:45:37.637645 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633492    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:45:37.637645 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633494    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:45:37.637645 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633497    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:45:37.637645 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633499    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:45:37.637645 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633503    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:45:37.638097 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633505    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:45:37.638097 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633508    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:45:37.638097 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633510    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:45:37.638097 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633514    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:45:37.638097 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633516    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:45:37.638097 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633519    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:45:37.638097 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633521    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:45:37.638097 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633524    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:45:37.638097 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633527    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:45:37.638097 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633529    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:45:37.638097 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633532    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:45:37.638097 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633535    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:45:37.638097 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633537    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:45:37.638097 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633542    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:45:37.638097 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633546    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:45:37.638097 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633548    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:45:37.638097 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633552    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:45:37.638097 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633554    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:45:37.638097 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633557    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:45:37.638535 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633559    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:45:37.638535 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633562    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:45:37.638535 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633564    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:45:37.638535 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633567    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:45:37.638535 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633569    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:45:37.638535 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633572    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:45:37.638535 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633574    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:45:37.638535 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633578    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:45:37.638535 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633580    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:45:37.638535 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633583    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:45:37.638535 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633585    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:45:37.638535 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633588    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:45:37.638535 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633590    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:45:37.638535 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633596    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:45:37.638535 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633599    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:45:37.638535 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633611    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:45:37.638535 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633614    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:45:37.638535 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633617    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:45:37.638535 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633619    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:45:37.638535 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633622    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:45:37.639086 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633624    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:45:37.639086 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633627    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:45:37.639086 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633630    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:45:37.639086 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633632    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:45:37.639086 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.633635    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:45:37.639086 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634638    2579 flags.go:64] FLAG: --address="0.0.0.0"
Oct 29 21:45:37.639086 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634648    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 29 21:45:37.639086 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634653    2579 flags.go:64] FLAG: --anonymous-auth="true"
Oct 29 21:45:37.639086 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634658    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 29 21:45:37.639086 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634662    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 29 21:45:37.639086 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634666    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 29 21:45:37.639086 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634670    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 29 21:45:37.639086 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634674    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 29 21:45:37.639086 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634677    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 29 21:45:37.639086 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634680    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 29 21:45:37.639086 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634684    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 29 21:45:37.639086 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634687    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 29 21:45:37.639086 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634690    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 29 21:45:37.639086 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634693    2579 flags.go:64] FLAG: --cgroup-root=""
Oct 29 21:45:37.639086 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634696    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 29 21:45:37.639086 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634699    2579 flags.go:64] FLAG: --client-ca-file=""
Oct 29 21:45:37.639086 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634702    2579 flags.go:64] FLAG: --cloud-provider="external"
Oct 29 21:45:37.639086 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634705    2579 flags.go:64] FLAG: --cluster-dns="[]"
Oct 29 21:45:37.639625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634710    2579 flags.go:64] FLAG: --cluster-domain=""
Oct 29 21:45:37.639625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634713    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 29 21:45:37.639625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634716    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 29 21:45:37.639625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634720    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 29 21:45:37.639625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634724    2579 flags.go:64] FLAG: --container-log-max-files="5"
Oct 29 21:45:37.639625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634728    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 29 21:45:37.639625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634731    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 29 21:45:37.639625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634734    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 29 21:45:37.639625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634737    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 29 21:45:37.639625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634740    2579 flags.go:64] FLAG: --contention-profiling="false"
Oct 29 21:45:37.639625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634743    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 29 21:45:37.639625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634747    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 29 21:45:37.639625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634750    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 29 21:45:37.639625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634753    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 29 21:45:37.639625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634757    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 29 21:45:37.639625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634761    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 29 21:45:37.639625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634764    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 29 21:45:37.639625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634767    2579 flags.go:64] FLAG: --enable-load-reader="false"
Oct 29 21:45:37.639625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634770    2579 flags.go:64] FLAG: --enable-server="true"
Oct 29 21:45:37.639625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634773    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 29 21:45:37.639625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634777    2579 flags.go:64] FLAG: --event-burst="100"
Oct 29 21:45:37.639625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634780    2579 flags.go:64] FLAG: --event-qps="50"
Oct 29 21:45:37.639625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634784    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 29 21:45:37.639625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634787    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 29 21:45:37.640215 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634790    2579 flags.go:64] FLAG: --eviction-hard=""
Oct 29 21:45:37.640215 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634794    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 29 21:45:37.640215 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634797    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 29 21:45:37.640215 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634800    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 29 21:45:37.640215 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634804    2579 flags.go:64] FLAG: --eviction-soft=""
Oct 29 21:45:37.640215 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634807    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 29 21:45:37.640215 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634810    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 29 21:45:37.640215 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634813    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 29 21:45:37.640215 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634816    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 29 21:45:37.640215 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634819    2579 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 29 21:45:37.640215 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634822    2579 flags.go:64] FLAG: --fail-swap-on="true"
Oct 29 21:45:37.640215 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634825    2579 flags.go:64] FLAG: --feature-gates=""
Oct 29 21:45:37.640215 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634829    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 29 21:45:37.640215 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634833    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 29 21:45:37.640215 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634836    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 29 21:45:37.640215 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634839    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 29 21:45:37.640215 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634842    2579 flags.go:64] FLAG: --healthz-port="10248"
Oct 29 21:45:37.640215 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634845    2579 flags.go:64] FLAG: --help="false"
Oct 29 21:45:37.640215 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634848    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-129-146.ec2.internal"
Oct 29 21:45:37.640215 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634851    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 29 21:45:37.640215 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634854    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 29 21:45:37.640215 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634857    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 29 21:45:37.640215 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634861    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 29 21:45:37.640215 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634864    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634867    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634870    2579 flags.go:64] FLAG: --image-service-endpoint=""
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634873    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634876    2579 flags.go:64] FLAG: --kube-api-burst="100"
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634879    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634882    2579 flags.go:64] FLAG: --kube-api-qps="50"
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634885    2579 flags.go:64] FLAG: --kube-reserved=""
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634888    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634891    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634894    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634896    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634899    2579 flags.go:64] FLAG: --lock-file=""
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634904    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634907    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634910    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634917    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634920    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634923    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634926    2579 flags.go:64] FLAG: --logging-format="text"
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634929    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634932    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634936    2579 flags.go:64] FLAG: --manifest-url=""
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634938    2579 flags.go:64] FLAG: --manifest-url-header=""
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634943    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 29 21:45:37.640780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634946    2579 flags.go:64] FLAG: --max-open-files="1000000"
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634950    2579 flags.go:64] FLAG: --max-pods="110"
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634953    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634956    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634959    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634962    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634965    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634968    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634971    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634978    2579 flags.go:64] FLAG: --node-status-max-images="50"
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634982    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634986    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634989    2579 flags.go:64] FLAG: --pod-cidr=""
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634992    2579 flags.go:64] FLAG: --pod-manifest-path=""
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634995    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.634998    2579 flags.go:64] FLAG: --pods-per-core="0"
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635001    2579 flags.go:64] FLAG: --port="10250"
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635004    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635007    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-038c5f85302bf0b24"
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635011    2579 flags.go:64] FLAG: --qos-reserved=""
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635015    2579 flags.go:64] FLAG: --read-only-port="10255"
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635018    2579 flags.go:64] FLAG: --register-node="true"
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635021    2579 flags.go:64] FLAG: --register-with-taints=""
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635027    2579 flags.go:64] FLAG: --registry-burst="10"
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635030    2579 flags.go:64] FLAG: --registry-qps="5"
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635033    2579 flags.go:64] FLAG: --reserved-cpus=""
Oct 29 21:45:37.641425 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635046    2579 flags.go:64] FLAG: --reserved-memory=""
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635050    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635053    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635056    2579 flags.go:64] FLAG: --rotate-certificates="false"
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635060    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635063    2579 flags.go:64] FLAG: --runonce="false"
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635066    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635069    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635072    2579 flags.go:64] FLAG: --seccomp-default="false"
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635075    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635078    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635081    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635084    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635087    2579 flags.go:64] FLAG: --storage-driver-password="root"
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635090    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635093    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635096    2579 flags.go:64] FLAG: --storage-driver-user="root"
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635098    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635102    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635105    2579 flags.go:64] FLAG: --system-cgroups=""
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635107    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635113    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635116    2579 flags.go:64] FLAG: --tls-cert-file=""
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635118    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635123    2579 flags.go:64] FLAG: --tls-min-version=""
Oct 29 21:45:37.642062 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635126    2579 flags.go:64] FLAG: --tls-private-key-file=""
Oct 29 21:45:37.642650 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635131    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 29 21:45:37.642650 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635134    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 29 21:45:37.642650 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635137    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 29 21:45:37.642650 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635141    2579 flags.go:64] FLAG: --v="2"
Oct 29 21:45:37.642650 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635145    2579 flags.go:64] FLAG: --version="false"
Oct 29 21:45:37.642650 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635149    2579 flags.go:64] FLAG: --vmodule=""
Oct 29 21:45:37.642650 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635153    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 29 21:45:37.642650 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635156    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 29 21:45:37.642650 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635299    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:45:37.642650 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635303    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:45:37.642650 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635307    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:45:37.642650 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635310    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:45:37.642650 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635313    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:45:37.642650 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635316    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:45:37.642650 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635319    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:45:37.642650 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635322    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:45:37.642650 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635325    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:45:37.642650 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635327    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:45:37.642650 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635330    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:45:37.642650 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635333    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:45:37.642650 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635335    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:45:37.643165 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635338    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:45:37.643165 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635340    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:45:37.643165 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635343    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:45:37.643165 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635346    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:45:37.643165 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635349    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:45:37.643165 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635351    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:45:37.643165 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635354    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:45:37.643165 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635357    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:45:37.643165 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635359    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:45:37.643165 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635362    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:45:37.643165 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635365    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:45:37.643165 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635369    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:45:37.643165 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635371    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:45:37.643165 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635374    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:45:37.643165 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635378    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:45:37.643165 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635380    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:45:37.643165 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635383    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:45:37.643165 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635386    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:45:37.643165 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635389    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:45:37.643165 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635391    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:45:37.643636 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635394    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:45:37.643636 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635397    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:45:37.643636 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635399    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:45:37.643636 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635402    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:45:37.643636 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635405    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:45:37.643636 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635408    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:45:37.643636 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635410    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:45:37.643636 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635413    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:45:37.643636 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635416    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:45:37.643636 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635418    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:45:37.643636 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635421    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:45:37.643636 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635424    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:45:37.643636 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635426    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:45:37.643636 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635429    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:45:37.643636 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635431    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:45:37.643636 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635434    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:45:37.643636 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635436    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:45:37.643636 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635439    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:45:37.643636 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635442    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:45:37.643636 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635444    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:45:37.644143 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635447    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:45:37.644143 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635450    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:45:37.644143 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635453    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:45:37.644143 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635457    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:45:37.644143 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635459    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:45:37.644143 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635462    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:45:37.644143 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635466    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:45:37.644143 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635468    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:45:37.644143 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635471    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:45:37.644143 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635474    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:45:37.644143 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635476    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:45:37.644143 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635479    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:45:37.644143 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635481    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:45:37.644143 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635484    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:45:37.644143 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635487    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:45:37.644143 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635489    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:45:37.644143 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635492    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:45:37.644143 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635495    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:45:37.644143 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635498    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:45:37.644590 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635502    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:45:37.644590 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635506    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:45:37.644590 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635509    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:45:37.644590 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635512    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:45:37.644590 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635515    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:45:37.644590 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635518    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:45:37.644590 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635520    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:45:37.644590 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635523    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:45:37.644590 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635525    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:45:37.644590 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635528    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:45:37.644590 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635531    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:45:37.644590 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635533    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:45:37.644590 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635536    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:45:37.644590 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635538    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:45:37.644590 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635541    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:45:37.644590 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635545    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:45:37.644590 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635550    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:45:37.644590 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635553    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:45:37.644590 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635556    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:45:37.645050 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635560    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:45:37.645050 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635563    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:45:37.645050 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635565    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:45:37.645050 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635568    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:45:37.645050 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635571    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:45:37.645050 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635573    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:45:37.645050 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635576    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:45:37.645050 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635578    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:45:37.645050 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635581    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:45:37.645050 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635583    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:45:37.645050 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635586    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:45:37.645050 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635589    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:45:37.645050 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635592    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:45:37.645050 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635594    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:45:37.645050 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635597    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:45:37.645050 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.635622    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 21:45:37.645432 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635660    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:45:37.645432 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635664    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:45:37.645432 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635667    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:45:37.645432 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635670    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:45:37.645432 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635673    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:45:37.645432 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635675    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:45:37.645432 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635678    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:45:37.645432 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635680    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:45:37.645432 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635683    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:45:37.645432 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635686    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:45:37.645432 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635689    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:45:37.645432 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635691    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:45:37.645432 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635694    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:45:37.645432 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635697    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:45:37.645432 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635701    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:45:37.645432 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635704    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:45:37.645432 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635708    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:45:37.645432 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635711    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:45:37.645432 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635714    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:45:37.645432 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635716    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:45:37.645910 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635719    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:45:37.645910 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635721    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:45:37.645910 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635724    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:45:37.645910 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635727    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:45:37.645910 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635729    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:45:37.645910 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635732    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:45:37.645910 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635734    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:45:37.645910 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635737    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:45:37.645910 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635740    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:45:37.645910 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635743    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:45:37.645910 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635745    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:45:37.645910 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635748    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:45:37.645910 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635750    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:45:37.645910 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635753    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:45:37.645910 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635756    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:45:37.645910 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635758    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:45:37.645910 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635761    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:45:37.645910 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635764    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:45:37.645910 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635767    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:45:37.645910 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635769    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:45:37.646411 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635772    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:45:37.646411 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635775    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:45:37.646411 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635778    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:45:37.646411 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635781    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:45:37.646411 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635784    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:45:37.646411 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635786    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:45:37.646411 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635790    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:45:37.646411 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635793    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:45:37.646411 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635797    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:45:37.646411 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635800    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:45:37.646411 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635802    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:45:37.646411 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635806    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:45:37.646411 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635808    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:45:37.646411 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635811    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:45:37.646411 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635813    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:45:37.646411 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635816    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:45:37.646411 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635819    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:45:37.646411 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635821    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:45:37.646411 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635824    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:45:37.646851 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635826    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:45:37.646851 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635829    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:45:37.646851 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635832    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:45:37.646851 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635835    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:45:37.646851 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635837    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:45:37.646851 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635840    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:45:37.646851 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635842    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:45:37.646851 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635845    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:45:37.646851 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635848    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:45:37.646851 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635852    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:45:37.646851 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635855    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:45:37.646851 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635858    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:45:37.646851 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635861    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:45:37.646851 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635864    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:45:37.646851 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635866    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:45:37.646851 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635869    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:45:37.646851 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635872    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:45:37.646851 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635874    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:45:37.646851 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635877    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:45:37.647296 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635881    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:45:37.647296 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635884    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:45:37.647296 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635887    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:45:37.647296 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635889    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:45:37.647296 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635892    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:45:37.647296 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635896    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:45:37.647296 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635900    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:45:37.647296 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635903    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:45:37.647296 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635906    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:45:37.647296 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635908    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:45:37.647296 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635911    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:45:37.647296 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635913    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:45:37.647296 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635916    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:45:37.647296 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635918    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:45:37.647296 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635921    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:45:37.647296 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635923    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:45:37.647296 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635926    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:45:37.647296 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635929    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:45:37.647296 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635932    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:45:37.647736 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635935    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:45:37.647736 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635937    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:45:37.647736 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635940    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:45:37.647736 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635942    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:45:37.647736 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635945    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:45:37.647736 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635948    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:45:37.647736 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635950    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:45:37.647736 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635954    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:45:37.647736 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.635957    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:45:37.647736 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.642671    2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 29 21:45:37.647736 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.642687    2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 29 21:45:37.647736 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.642700    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 29 21:45:37.647736 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.642704    2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 29 21:45:37.647736 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642760    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:45:37.647736 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642765    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:45:37.647736 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642768    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:45:37.647736 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642771    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:45:37.647736 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642774    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:45:37.647736 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642777    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:45:37.648293 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642779    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:45:37.648293 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642782    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:45:37.648293 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642785    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:45:37.648293 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642788    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:45:37.648293 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642790    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:45:37.648293 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642793    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:45:37.648293 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642795    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:45:37.648293 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642798    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:45:37.648293 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642801    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:45:37.648293 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642804    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:45:37.648293 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642807    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:45:37.648293 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642810    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:45:37.648293 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642813    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:45:37.648293 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642816    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:45:37.648293 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642818    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:45:37.648293 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642822    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:45:37.648293 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642826    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:45:37.648293 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642829    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:45:37.648293 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642832    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:45:37.648293 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642834    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:45:37.648753 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642837    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:45:37.648753 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642839    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:45:37.648753 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642842    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:45:37.648753 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642846    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:45:37.648753 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642849    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:45:37.648753 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642852    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:45:37.648753 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642855    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:45:37.648753 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642857    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:45:37.648753 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642861    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:45:37.648753 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642863    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:45:37.648753 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642866    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:45:37.648753 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642869    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:45:37.648753 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642871    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:45:37.648753 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642874    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:45:37.648753 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642876    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:45:37.648753 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642879    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:45:37.648753 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642882    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:45:37.648753 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642884    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:45:37.648753 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642887    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:45:37.648753 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642890    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:45:37.649223 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642892    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:45:37.649223 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642895    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:45:37.649223 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642897    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:45:37.649223 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642900    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:45:37.649223 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642902    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:45:37.649223 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642905    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:45:37.649223 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642908    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:45:37.649223 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642910    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:45:37.649223 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642913    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:45:37.649223 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642915    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:45:37.649223 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642918    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:45:37.649223 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642920    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:45:37.649223 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642923    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:45:37.649223 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642926    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:45:37.649223 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642928    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:45:37.649223 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642932    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:45:37.649223 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642935    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:45:37.649223 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642938    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:45:37.649223 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642940    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:45:37.649223 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642943    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:45:37.649706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642945    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:45:37.649706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642948    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:45:37.649706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642951    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:45:37.649706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642954    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:45:37.649706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642956    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:45:37.649706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642959    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:45:37.649706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642962    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:45:37.649706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642964    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:45:37.649706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642967    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:45:37.649706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642970    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:45:37.649706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642972    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:45:37.649706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642975    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:45:37.649706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642978    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:45:37.649706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642981    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:45:37.649706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642983    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:45:37.649706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642985    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:45:37.649706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642988    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:45:37.649706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642991    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:45:37.649706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642994    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:45:37.650154 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.642998    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:45:37.650154 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643002    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:45:37.650154 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643005    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:45:37.650154 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643007    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:45:37.650154 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643010    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:45:37.650154 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643013    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:45:37.650154 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643015    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:45:37.650154 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643018    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:45:37.650154 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643021    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:45:37.650154 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643024    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:45:37.650154 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643027    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:45:37.650154 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643030    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:45:37.650154 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643033    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:45:37.650154 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643035    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:45:37.650154 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643053    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:45:37.650154 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643057    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:45:37.650154 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643059    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:45:37.650154 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643062    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:45:37.650154 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643065    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:45:37.650592 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643067    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:45:37.650592 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643070    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:45:37.650592 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.643098    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 21:45:37.650592 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643252    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:45:37.650592 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643256    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:45:37.650592 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643259    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:45:37.650592 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643262    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:45:37.650592 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643264    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:45:37.650592 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643267    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:45:37.650592 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643270    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:45:37.650592 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643272    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:45:37.650592 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643275    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:45:37.650592 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643278    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:45:37.650592 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643280    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:45:37.650592 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643283    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:45:37.650592 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643285    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:45:37.651021 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643288    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:45:37.651021 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643290    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:45:37.651021 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643293    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:45:37.651021 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643295    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:45:37.651021 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643298    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:45:37.651021 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643301    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:45:37.651021 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643304    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:45:37.651021 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643306    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:45:37.651021 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643310    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:45:37.651021 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643313    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:45:37.651021 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643315    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:45:37.651021 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643318    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:45:37.651021 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643320    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:45:37.651021 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643323    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:45:37.651021 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643325    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:45:37.651021 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643328    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:45:37.651021 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643330    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:45:37.651021 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643333    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:45:37.651021 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643335    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:45:37.651501 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643338    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:45:37.651501 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643340    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:45:37.651501 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643343    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:45:37.651501 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643345    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:45:37.651501 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643348    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:45:37.651501 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643350    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:45:37.651501 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643353    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:45:37.651501 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643355    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:45:37.651501 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643358    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:45:37.651501 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643360    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:45:37.651501 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643363    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:45:37.651501 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643365    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:45:37.651501 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643368    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:45:37.651501 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643370    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:45:37.651501 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643373    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:45:37.651501 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643375    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:45:37.651501 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643378    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:45:37.651501 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643380    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:45:37.651501 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643383    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:45:37.651501 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643386    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:45:37.652131 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643388    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:45:37.652131 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643391    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:45:37.652131 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643394    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:45:37.652131 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643397    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:45:37.652131 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643399    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:45:37.652131 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643402    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:45:37.652131 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643404    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:45:37.652131 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643407    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:45:37.652131 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643409    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:45:37.652131 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643412    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:45:37.652131 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643414    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:45:37.652131 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643416    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:45:37.652131 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643419    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:45:37.652131 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643422    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:45:37.652131 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643424    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:45:37.652131 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643426    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:45:37.652131 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643429    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:45:37.652131 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643431    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:45:37.652131 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643434    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:45:37.652666 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643436    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:45:37.652666 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643439    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:45:37.652666 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643441    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:45:37.652666 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643444    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:45:37.652666 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643447    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:45:37.652666 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643451    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:45:37.652666 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643454    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:45:37.652666 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643457    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:45:37.652666 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643459    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:45:37.652666 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643462    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:45:37.652666 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643465    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:45:37.652666 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643469    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:45:37.652666 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643473    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:45:37.652666 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643475    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:45:37.652666 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643478    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:45:37.652666 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643481    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:45:37.652666 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643484    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:45:37.652666 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643487    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:45:37.652666 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643489    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:45:37.653444 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643492    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:45:37.653444 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643494    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:45:37.653444 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643497    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:45:37.653444 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643499    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:45:37.653444 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643502    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:45:37.653444 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643504    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:45:37.653444 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643507    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:45:37.653444 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643509    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:45:37.653444 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643512    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:45:37.653444 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643514    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:45:37.653444 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643517    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:45:37.653444 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643519    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:45:37.653444 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643522    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:45:37.653444 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643524    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:45:37.653444 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643526    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:45:37.653444 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643529    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:45:37.653925 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.643551    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 21:45:37.653925 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643582    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:45:37.653925 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643586    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:45:37.653925 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643588    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:45:37.653925 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643591    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:45:37.653925 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643594    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:45:37.653925 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643597    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:45:37.653925 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643600    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:45:37.653925 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643603    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:45:37.653925 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643605    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:45:37.653925 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643607    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:45:37.653925 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643611    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:45:37.653925 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643614    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:45:37.653925 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643616    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:45:37.653925 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643619    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:45:37.653925 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643622    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:45:37.654323 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643624    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:45:37.654323 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643627    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:45:37.654323 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643629    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:45:37.654323 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643632    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:45:37.654323 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643635    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:45:37.654323 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643637    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:45:37.654323 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643640    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:45:37.654323 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643643    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:45:37.654323 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643645    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:45:37.654323 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643648    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:45:37.654323 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643651    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:45:37.654323 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643653    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:45:37.654323 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643656    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:45:37.654323 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643658    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:45:37.654323 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643661    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:45:37.654323 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643663    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:45:37.654323 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643666    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:45:37.654323 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643668    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:45:37.654323 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643671    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:45:37.654323 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643673    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:45:37.654774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643676    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:45:37.654774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643678    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:45:37.654774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643681    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:45:37.654774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643684    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:45:37.654774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643687    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:45:37.654774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643689    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:45:37.654774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643692    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:45:37.654774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643694    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:45:37.654774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643697    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:45:37.654774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643699    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:45:37.654774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643702    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:45:37.654774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643706    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:45:37.654774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643709    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:45:37.654774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643712    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:45:37.654774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643715    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:45:37.654774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643718    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:45:37.654774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643720    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:45:37.654774 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643723    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:45:37.655240 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643727    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:45:37.655240 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643733    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:45:37.655240 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643736    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:45:37.655240 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643740    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:45:37.655240 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643743    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:45:37.655240 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643746    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:45:37.655240 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643748    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:45:37.655240 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643751    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:45:37.655240 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643753    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:45:37.655240 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643756    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:45:37.655240 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643758    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:45:37.655240 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643761    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:45:37.655240 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643763    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:45:37.655240 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643766    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:45:37.655240 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643769    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:45:37.655240 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643771    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:45:37.655240 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643774    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:45:37.655240 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643777    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:45:37.655240 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643779    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:45:37.655240 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643782    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:45:37.655706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643785    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:45:37.655706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643787    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:45:37.655706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643790    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:45:37.655706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643792    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:45:37.655706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643795    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:45:37.655706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643798    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:45:37.655706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643801    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:45:37.655706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643803    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:45:37.655706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643806    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:45:37.655706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643808    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:45:37.655706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643811    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:45:37.655706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643814    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:45:37.655706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643816    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:45:37.655706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643820    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:45:37.655706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643823    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:45:37.655706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643826    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:45:37.655706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643828    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:45:37.655706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643831    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:45:37.655706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643833    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:45:37.655706 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643835    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:45:37.656181 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643838    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:45:37.656181 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643841    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:45:37.656181 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643843    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:45:37.656181 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643846    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:45:37.656181 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643848    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:45:37.656181 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643851    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:45:37.656181 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643854    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:45:37.656181 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643856    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:45:37.656181 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643859    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:45:37.656181 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643861    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:45:37.656181 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643864    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:45:37.656181 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643866    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:45:37.656181 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:37.643869    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:45:37.656181 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.643935    2579 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 29 21:45:37.656181 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.646307    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 29 21:45:37.656181 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.647368    2579 server.go:1014] "Starting client certificate rotation"
Oct 29 21:45:37.656181 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.647455    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 21:45:37.656181 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.648181    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 21:45:37.656644 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.649536    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 21:45:37.656644 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.653724    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 21:45:37.663691 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.663673    2579 log.go:25] "Validated CRI v1 runtime API"
Oct 29 21:45:37.666759 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.666744    2579 log.go:25] "Validated CRI v1 image API"
Oct 29 21:45:37.667551 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.667529    2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 29 21:45:37.670914 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.670892    2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c05f9a57-5ec0-4474-89ca-a39690efa53e:/dev/nvme0n1p3 df845650-e347-4498-a9f0-d5dc0ae2439a:/dev/nvme0n1p4]
Oct 29 21:45:37.670989 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.670913    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 29 21:45:37.676820 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.676701    2579 manager.go:217] Machine: {Timestamp:2026-10-29 21:45:37.675290316 +0000 UTC m=+0.368167244 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099976 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22465b80dda8f7b658a3ae5cef31e2 SystemUUID:ec22465b-80dd-a8f7-b658-a3ae5cef31e2 BootID:fbea307b-1efb-4aea-8948-138de3188448 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c2:55:79:e1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c2:55:79:e1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:ac:61:73:62:e3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 29 21:45:37.677443 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.677431    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 29 21:45:37.677570 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.677549    2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 29 21:45:37.681309 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.681284    2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 21:45:37.682153 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.681311    2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-146.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 29 21:45:37.682906 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.682897    2579 topology_manager.go:143] "Creating topology manager with none policy"
Oct 29 21:45:37.682941 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.682909    2579 container_manager_linux.go:311] "Creating device plugin manager"
Oct 29 21:45:37.682941 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.682934    2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 21:45:37.682998 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.682946    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 21:45:37.684216 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.684206    2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 29 21:45:37.688807 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.688795    2579 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 29 21:45:37.688931 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.688921    2579 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 29 21:45:37.689788 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.689778    2579 kubelet.go:493] "Attempting to sync node with API server"
Oct 29 21:45:37.689822 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.689791    2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 29 21:45:37.689822 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.689802    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 29 21:45:37.689822 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.689812    2579 kubelet.go:405] "Adding apiserver pod source"
Oct 29 21:45:37.689822 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.689820    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 21:45:37.690210 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.690198    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 21:45:37.692808 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.692789    2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 29 21:45:37.700145 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.700129    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 21:45:37.700954 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.700942    2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 29 21:45:37.701002 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.700972    2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 29 21:45:37.701146 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.701136    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 29 21:45:37.701177 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.701152    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 29 21:45:37.701177 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.701161    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 29 21:45:37.701177 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.701169    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 29 21:45:37.701177 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.701175    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 29 21:45:37.701788 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.701777    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 29 21:45:37.701894 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.701881    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 29 21:45:37.701944 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.701901    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 29 21:45:37.701944 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.701916    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 29 21:45:37.701944 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.701927    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 29 21:45:37.702086 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.701963    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 29 21:45:37.702086 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.701982    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 29 21:45:37.703670 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.703648    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 29 21:45:37.703727 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.703719    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 29 21:45:37.708523 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.708504    2579 server.go:1263] "Started kubelet"
Oct 29 21:45:37.708608 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.708580    2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 21:45:37.708752 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.708712    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 29 21:45:37.708816 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.708769    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 29 21:45:37.709258 ip-10-0-129-146 systemd[1]: Started Kubernetes Kubelet.
Oct 29 21:45:37.710238 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.710220    2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 29 21:45:37.711518 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.711507    2579 server.go:317] "Adding debug handlers to kubelet server"
Oct 29 21:45:37.715226 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:37.715209    2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 29 21:45:37.717579 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.717564    2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 29 21:45:37.717753 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:37.717691    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-146.ec2.internal\" not found"
Oct 29 21:45:37.717875 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.717569    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 29 21:45:37.718948 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.718926    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 29 21:45:37.719871 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.719852    2579 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 29 21:45:37.719871 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.719870    2579 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 29 21:45:37.719993 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.719967    2579 reconstruct.go:97] "Volume reconstruction finished"
Oct 29 21:45:37.719993 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.719978    2579 reconciler.go:29] "Reconciler: start to sync state"
Oct 29 21:45:37.720715 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.720699    2579 factory.go:153] Registering CRI-O factory
Oct 29 21:45:37.720715 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.720718    2579 factory.go:223] Registration of the crio container factory successfully
Oct 29 21:45:37.720839 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.720768    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 29 21:45:37.720839 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.720779    2579 factory.go:55] Registering systemd factory
Oct 29 21:45:37.720839 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.720787    2579 factory.go:223] Registration of the systemd container factory successfully
Oct 29 21:45:37.720839 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.720807    2579 factory.go:103] Registering Raw factory
Oct 29 21:45:37.720839 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.720820    2579 manager.go:1199] Started watching for new ooms in manager
Oct 29 21:45:37.721853 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.721839    2579 manager.go:319] Starting recovery of all containers
Oct 29 21:45:37.735285 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.735267    2579 manager.go:324] Recovery completed
Oct 29 21:45:37.739380 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.739366    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:45:37.739467 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.739443    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-146.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 29 21:45:37.739467 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:37.739450    2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-146.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 29 21:45:37.740606 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:37.739446    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-146.ec2.internal.18e31e4033263e6a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-146.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-146.ec2.internal,},FirstTimestamp:2026-10-29 21:45:37.708473962 +0000 UTC m=+0.401350889,LastTimestamp:2026-10-29 21:45:37.708473962 +0000 UTC m=+0.401350889,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-146.ec2.internal,}"
Oct 29 21:45:37.741113 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.741095    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:45:37.741186 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.741123    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:45:37.741186 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.741137    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:45:37.741260 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.741243    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 29 21:45:37.741735 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.741720    2579 cpu_manager.go:226] "Starting" policy="none"
Oct 29 21:45:37.741735 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.741734    2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 29 21:45:37.741853 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.741752    2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 29 21:45:37.741853 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.741785    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:45:37.742466 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.742448    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:45:37.742536 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.742477    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:45:37.742536 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.742493    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:45:37.757797 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:37.757713    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-146.ec2.internal.18e31e4035184359  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-146.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-146.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-146.ec2.internal,},FirstTimestamp:2026-10-29 21:45:37.741112153 +0000 UTC m=+0.433989080,LastTimestamp:2026-10-29 21:45:37.741112153 +0000 UTC m=+0.433989080,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-146.ec2.internal,}"
Oct 29 21:45:37.758556 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:37.758536    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 29 21:45:37.758609 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:37.758536    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-146.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 29 21:45:37.772145 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:37.772123    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 29 21:45:37.772581 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:37.772513    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-146.ec2.internal.18e31e40351885f6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-146.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-146.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-146.ec2.internal,},FirstTimestamp:2026-10-29 21:45:37.741129206 +0000 UTC m=+0.434006141,LastTimestamp:2026-10-29 21:45:37.741129206 +0000 UTC m=+0.434006141,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-146.ec2.internal,}"
Oct 29 21:45:37.774128 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.774112    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7nfzv"
Oct 29 21:45:37.787793 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.787774    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7nfzv"
Oct 29 21:45:37.791973 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:37.791881    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-146.ec2.internal.18e31e403518bd22  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-146.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-146.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-146.ec2.internal,},FirstTimestamp:2026-10-29 21:45:37.74114333 +0000 UTC m=+0.434020264,LastTimestamp:2026-10-29 21:45:37.74114333 +0000 UTC m=+0.434020264,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-146.ec2.internal,}"
Oct 29 21:45:37.817798 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:37.817782    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-146.ec2.internal\" not found"
Oct 29 21:45:37.881126 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.881086    2579 policy_none.go:50] "Start"
Oct 29 21:45:37.881126 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.881103    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 29 21:45:37.881126 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.881114    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 29 21:45:37.883161 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.883151    2579 policy_none.go:44] "Start"
Oct 29 21:45:37.909985 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.907856    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 29 21:45:37.909985 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.909069    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 29 21:45:37.909985 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.909092    2579 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 29 21:45:37.909985 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.909112    2579 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 29 21:45:37.909985 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:37.909149    2579 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 29 21:45:37.915604 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.915571    2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 29 21:45:37.917797 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.917783    2579 manager.go:341] "Starting Device Plugin manager"
Oct 29 21:45:37.917894 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:37.917881    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-146.ec2.internal\" not found"
Oct 29 21:45:37.918474 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:37.918458    2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 29 21:45:37.918474 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.918470    2579 server.go:87] "Starting device plugin registration server"
Oct 29 21:45:37.918659 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.918647    2579 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 29 21:45:37.918718 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.918658    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 29 21:45:37.918770 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.918760    2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 29 21:45:37.918849 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.918834    2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 29 21:45:37.918849 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.918848    2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 29 21:45:37.919291 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:37.919273    2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 29 21:45:37.919389 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:37.919316    2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-146.ec2.internal\" not found"
Oct 29 21:45:37.923839 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:37.923825    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:45:37.968322 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:37.968305    2579 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-146.ec2.internal\" not found" node="ip-10-0-129-146.ec2.internal"
Oct 29 21:45:38.009442 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.009422    2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal"]
Oct 29 21:45:38.009495 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.009485    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:45:38.011240 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.011227    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:45:38.011291 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.011251    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:45:38.011291 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.011261    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:45:38.013475 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.013463    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:45:38.013625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.013611    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal"
Oct 29 21:45:38.013661 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.013639    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:45:38.014204 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.014190    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:45:38.014252 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.014218    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:45:38.014252 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.014228    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:45:38.014311 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.014194    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:45:38.014311 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.014298    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:45:38.014311 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.014309    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:45:38.016399 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.016384    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal"
Oct 29 21:45:38.016471 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.016407    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:45:38.018811 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.018795    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:45:38.019865 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.019849    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:45:38.019947 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.019879    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:45:38.019947 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.019889    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:45:38.019947 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.019916    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:45:38.019947 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.019931    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:45:38.019947 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.019894    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:45:38.020159 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.019960    2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-146.ec2.internal"
Oct 29 21:45:38.022293 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.022280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/493b35d9c8008ff0c00d95f4c3e7e953-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal\" (UID: \"493b35d9c8008ff0c00d95f4c3e7e953\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal"
Oct 29 21:45:38.022337 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.022302    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/493b35d9c8008ff0c00d95f4c3e7e953-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal\" (UID: \"493b35d9c8008ff0c00d95f4c3e7e953\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal"
Oct 29 21:45:38.022337 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.022327    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4db57e6b6465b77567369446b9e61568-config\") pod \"kube-apiserver-proxy-ip-10-0-129-146.ec2.internal\" (UID: \"4db57e6b6465b77567369446b9e61568\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal"
Oct 29 21:45:38.040474 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:38.040452    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-146.ec2.internal\" not found" node="ip-10-0-129-146.ec2.internal"
Oct 29 21:45:38.041057 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.041030    2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-146.ec2.internal"
Oct 29 21:45:38.041137 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:38.041063    2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-146.ec2.internal\": node \"ip-10-0-129-146.ec2.internal\" not found"
Oct 29 21:45:38.044726 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:38.044712    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-146.ec2.internal\" not found" node="ip-10-0-129-146.ec2.internal"
Oct 29 21:45:38.073613 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:38.073586    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-146.ec2.internal\" not found"
Oct 29 21:45:38.122866 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.122846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/493b35d9c8008ff0c00d95f4c3e7e953-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal\" (UID: \"493b35d9c8008ff0c00d95f4c3e7e953\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal"
Oct 29 21:45:38.122938 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.122877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/493b35d9c8008ff0c00d95f4c3e7e953-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal\" (UID: \"493b35d9c8008ff0c00d95f4c3e7e953\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal"
Oct 29 21:45:38.122938 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.122901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4db57e6b6465b77567369446b9e61568-config\") pod \"kube-apiserver-proxy-ip-10-0-129-146.ec2.internal\" (UID: \"4db57e6b6465b77567369446b9e61568\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal"
Oct 29 21:45:38.123004 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.122953    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/493b35d9c8008ff0c00d95f4c3e7e953-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal\" (UID: \"493b35d9c8008ff0c00d95f4c3e7e953\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal"
Oct 29 21:45:38.123004 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.122956    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/493b35d9c8008ff0c00d95f4c3e7e953-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal\" (UID: \"493b35d9c8008ff0c00d95f4c3e7e953\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal"
Oct 29 21:45:38.123094 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.123013    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4db57e6b6465b77567369446b9e61568-config\") pod \"kube-apiserver-proxy-ip-10-0-129-146.ec2.internal\" (UID: \"4db57e6b6465b77567369446b9e61568\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal"
Oct 29 21:45:38.174329 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:38.174288    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-146.ec2.internal\" not found"
Oct 29 21:45:38.275263 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:38.275245    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-146.ec2.internal\" not found"
Oct 29 21:45:38.344932 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.344906    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal"
Oct 29 21:45:38.350074 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.350058    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal"
Oct 29 21:45:38.376085 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:38.376069    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-146.ec2.internal\" not found"
Oct 29 21:45:38.476604 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:38.476548    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-146.ec2.internal\" not found"
Oct 29 21:45:38.577193 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:38.577170    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-146.ec2.internal\" not found"
Oct 29 21:45:38.588582 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:38.588566    2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:45:40.217856 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.217840    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal"
Oct 29 21:45:40.245716 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.245699    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 21:45:40.246784 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.246769    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 29 21:45:40.246880 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.246863    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 29 21:45:40.246953 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.246918    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 29 21:45:40.247721 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.247708    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal"
Oct 29 21:45:40.289332 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.289315    2579 apiserver.go:52] "Watching apiserver"
Oct 29 21:45:40.317648 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.317633    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 29 21:45:40.333748 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.333700    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 21:45:40.344922 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.344903    2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 29 21:45:40.345214 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.345193    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-74bdq","kube-system/global-pull-secret-syncer-hwq7z","kube-system/konnectivity-agent-g2rdv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz","openshift-multus/network-metrics-daemon-l7tkk","openshift-network-diagnostics/network-check-target-pkmzk","openshift-network-operator/iptables-alerter-t997r","openshift-cluster-node-tuning-operator/tuned-s6mfd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal","openshift-multus/multus-additional-cni-plugins-lzjtw","openshift-multus/multus-ts2kk","openshift-dns/node-resolver-rrmxb","openshift-image-registry/node-ca-knxtf"]
Oct 29 21:45:40.347596 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.347575    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t997r"
Oct 29 21:45:40.350388 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.350369    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g2rdv"
Oct 29 21:45:40.352772 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.352755    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:45:40.355320 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.355300    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:45:40.355420 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:40.355383    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7tkk" podUID="0febea83-955d-4a56-8bc4-c883a4141109"
Oct 29 21:45:40.357778 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.357763    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.360006 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.359988    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.362682 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.362663    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:45:40.364188 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.364150    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:45:40.364344 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.364279    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 29 21:45:40.364414 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.364367    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 29 21:45:40.364414 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.364408    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8ld6j\""
Oct 29 21:45:40.364797 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.364783    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.366521 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.366507    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 29 21:45:40.366598 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.366549    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 29 21:45:40.366862 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.366846    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-l6fj9\""
Oct 29 21:45:40.367100 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.367086    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rrmxb"
Oct 29 21:45:40.368983 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.368966    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 29 21:45:40.369082 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.369033    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 29 21:45:40.369253 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.369239    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5fns5\""
Oct 29 21:45:40.369678 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.369658    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-knxtf"
Oct 29 21:45:40.372359 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.371591    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 29 21:45:40.373519 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.373502    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:45:40.373617 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:40.373566    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwq7z" podUID="676904dc-4d4f-4bd6-b380-357d5d9a29d4"
Oct 29 21:45:40.376429 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.376413    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal"]
Oct 29 21:45:40.376522 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.376478    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:45:40.376560 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:40.376520    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkmzk" podUID="f934ff4f-dadc-469b-a87e-16db95e3181f"
Oct 29 21:45:40.392845 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.392806    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 21:40:39 +0000 UTC" deadline="2028-05-15 09:27:50.080864077 +0000 UTC"
Oct 29 21:45:40.392845 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.392842    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13523h42m9.688024362s"
Oct 29 21:45:40.402037 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.402015    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 29 21:45:40.402120 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.402069    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fkkb6"
Oct 29 21:45:40.414433 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.414414    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fkkb6"
Oct 29 21:45:40.419587 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.419564    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 29 21:45:40.427404 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427378    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/676904dc-4d4f-4bd6-b380-357d5d9a29d4-kubelet-config\") pod \"global-pull-secret-syncer-hwq7z\" (UID: \"676904dc-4d4f-4bd6-b380-357d5d9a29d4\") " pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:45:40.427498 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427407    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-log-socket\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.427498 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/259e6b95-d64c-4101-871f-4a933d4ac38f-env-overrides\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.427498 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b546ae66-9568-4840-a4e5-659d93eb80c5-tmp\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.427627 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427498    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-multus-cni-dir\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.427627 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427523    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-cni-bin\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.427627 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427559    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-cni-netd\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.427627 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427582    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/259e6b95-d64c-4101-871f-4a933d4ac38f-ovnkube-script-lib\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.427742 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427629    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-var-lib-kubelet\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.427742 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427652    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-host\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.427742 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427670    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9400c4a1-ccf5-4b45-94f8-e37079762e4f-cni-binary-copy\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:45:40.427742 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427686    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc992647-89dd-44cc-ba19-3fdb03e125de-hosts-file\") pod \"node-resolver-rrmxb\" (UID: \"fc992647-89dd-44cc-ba19-3fdb03e125de\") " pod="openshift-dns/node-resolver-rrmxb"
Oct 29 21:45:40.427742 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427713    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-run-systemd\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.427742 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-systemd\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.427903 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9400c4a1-ccf5-4b45-94f8-e37079762e4f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:45:40.427903 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-cnibin\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.427903 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bhr9\" (UniqueName: \"kubernetes.io/projected/fc992647-89dd-44cc-ba19-3fdb03e125de-kube-api-access-9bhr9\") pod \"node-resolver-rrmxb\" (UID: \"fc992647-89dd-44cc-ba19-3fdb03e125de\") " pod="openshift-dns/node-resolver-rrmxb"
Oct 29 21:45:40.427903 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427821    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:45:40.427903 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427841    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-etc-selinux\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:45:40.427903 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427863    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-slash\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.427903 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-tuned\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.427903 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427896    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-etc-kubernetes\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.428125 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfnx5\" (UniqueName: \"kubernetes.io/projected/8428734b-4a9f-4df9-ac06-81074198a9ba-kube-api-access-vfnx5\") pod \"node-ca-knxtf\" (UID: \"8428734b-4a9f-4df9-ac06-81074198a9ba\") " pod="openshift-image-registry/node-ca-knxtf"
Oct 29 21:45:40.428125 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427938    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af572bfb-940b-4226-b91f-d06554b40d3c-konnectivity-ca\") pod \"konnectivity-agent-g2rdv\" (UID: \"af572bfb-940b-4226-b91f-d06554b40d3c\") " pod="kube-system/konnectivity-agent-g2rdv"
Oct 29 21:45:40.428125 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427963    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5mps\" (UniqueName: \"kubernetes.io/projected/259e6b95-d64c-4101-871f-4a933d4ac38f-kube-api-access-v5mps\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.428125 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.427986    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-sysctl-d\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.428125 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428010    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-sysctl-conf\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.428125 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbk9p\" (UniqueName: \"kubernetes.io/projected/9400c4a1-ccf5-4b45-94f8-e37079762e4f-kube-api-access-pbk9p\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:45:40.428125 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428043    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-registration-dir\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:45:40.428125 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428068    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/676904dc-4d4f-4bd6-b380-357d5d9a29d4-dbus\") pod \"global-pull-secret-syncer-hwq7z\" (UID: \"676904dc-4d4f-4bd6-b380-357d5d9a29d4\") " pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:45:40.428125 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428087    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brg9z\" (UniqueName: \"kubernetes.io/projected/f934ff4f-dadc-469b-a87e-16db95e3181f-kube-api-access-brg9z\") pod \"network-check-target-pkmzk\" (UID: \"f934ff4f-dadc-469b-a87e-16db95e3181f\") " pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:45:40.428125 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngw2c\" (UniqueName: \"kubernetes.io/projected/0febea83-955d-4a56-8bc4-c883a4141109-kube-api-access-ngw2c\") pod \"network-metrics-daemon-l7tkk\" (UID: \"0febea83-955d-4a56-8bc4-c883a4141109\") " pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:45:40.428125 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-sys\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.428453 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-system-cni-dir\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.428453 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428176    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-cni-binary-copy\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.428453 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428205    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-multus-socket-dir-parent\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.428453 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-hostroot\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.428453 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-multus-conf-dir\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.428453 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428270    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1953d5bb-b35f-4c91-a263-a3664cf7fab2-iptables-alerter-script\") pod \"iptables-alerter-t997r\" (UID: \"1953d5bb-b35f-4c91-a263-a3664cf7fab2\") " pod="openshift-network-operator/iptables-alerter-t997r"
Oct 29 21:45:40.428453 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428311    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-run-netns\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.428453 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428334    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-kubernetes\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.428453 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9400c4a1-ccf5-4b45-94f8-e37079762e4f-system-cni-dir\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:45:40.428453 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428368    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-run-netns\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.428453 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1953d5bb-b35f-4c91-a263-a3664cf7fab2-host-slash\") pod \"iptables-alerter-t997r\" (UID: \"1953d5bb-b35f-4c91-a263-a3664cf7fab2\") " pod="openshift-network-operator/iptables-alerter-t997r"
Oct 29 21:45:40.428453 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-socket-dir\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:45:40.428453 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-sys-fs\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:45:40.428858 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428484    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9400c4a1-ccf5-4b45-94f8-e37079762e4f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:45:40.428858 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9400c4a1-ccf5-4b45-94f8-e37079762e4f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:45:40.428858 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428533    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc992647-89dd-44cc-ba19-3fdb03e125de-tmp-dir\") pod \"node-resolver-rrmxb\" (UID: \"fc992647-89dd-44cc-ba19-3fdb03e125de\") " pod="openshift-dns/node-resolver-rrmxb"
Oct 29 21:45:40.428858 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs\") pod \"network-metrics-daemon-l7tkk\" (UID: \"0febea83-955d-4a56-8bc4-c883a4141109\") " pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:45:40.428858 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428588    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-run-k8s-cni-cncf-io\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.428858 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428606    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-systemd-units\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.428858 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428633    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-run-ovn\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.428858 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428650    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/259e6b95-d64c-4101-871f-4a933d4ac38f-ovnkube-config\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.428858 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428675    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-lib-modules\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.428858 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428690    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-var-lib-openvswitch\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.428858 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428718    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-etc-openvswitch\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.428858 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-modprobe-d\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.428858 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x75hw\" (UniqueName: \"kubernetes.io/projected/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-kube-api-access-x75hw\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.428858 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428779    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8428734b-4a9f-4df9-ac06-81074198a9ba-serviceca\") pod \"node-ca-knxtf\" (UID: \"8428734b-4a9f-4df9-ac06-81074198a9ba\") " pod="openshift-image-registry/node-ca-knxtf"
Oct 29 21:45:40.428858 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428807    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.428858 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-sysconfig\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.429276 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428868    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-var-lib-cni-multus\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.429276 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428895    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af572bfb-940b-4226-b91f-d06554b40d3c-agent-certs\") pod \"konnectivity-agent-g2rdv\" (UID: \"af572bfb-940b-4226-b91f-d06554b40d3c\") " pod="kube-system/konnectivity-agent-g2rdv"
Oct 29 21:45:40.429276 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428917    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-device-dir\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:45:40.429276 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428939    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/676904dc-4d4f-4bd6-b380-357d5d9a29d4-original-pull-secret\") pod \"global-pull-secret-syncer-hwq7z\" (UID: \"676904dc-4d4f-4bd6-b380-357d5d9a29d4\") " pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:45:40.429276 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-node-log\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.429276 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.428987    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-run-ovn-kubernetes\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.429276 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.429020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-run\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.429276 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.429051    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7dvt\" (UniqueName: \"kubernetes.io/projected/b546ae66-9568-4840-a4e5-659d93eb80c5-kube-api-access-x7dvt\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.429276 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.429074    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-run-multus-certs\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.429276 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.429104    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/259e6b95-d64c-4101-871f-4a933d4ac38f-ovn-node-metrics-cert\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.429276 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.429126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-var-lib-cni-bin\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.429276 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.429155    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-var-lib-kubelet\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.429276 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.429176    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62lfn\" (UniqueName: \"kubernetes.io/projected/ce551e36-0249-4723-b814-a98f0a282eed-kube-api-access-62lfn\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:45:40.429276 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.429200    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-kubelet\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.429276 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.429222    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-run-openvswitch\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.429276 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.429238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9400c4a1-ccf5-4b45-94f8-e37079762e4f-cnibin\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:45:40.430008 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.429268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-multus-daemon-config\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.430008 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.429300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8428734b-4a9f-4df9-ac06-81074198a9ba-host\") pod \"node-ca-knxtf\" (UID: \"8428734b-4a9f-4df9-ac06-81074198a9ba\") " pod="openshift-image-registry/node-ca-knxtf"
Oct 29 21:45:40.430008 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.429325    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kr4j\" (UniqueName: \"kubernetes.io/projected/1953d5bb-b35f-4c91-a263-a3664cf7fab2-kube-api-access-2kr4j\") pod \"iptables-alerter-t997r\" (UID: \"1953d5bb-b35f-4c91-a263-a3664cf7fab2\") " pod="openshift-network-operator/iptables-alerter-t997r"
Oct 29 21:45:40.465077 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.465057    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 29 21:45:40.465183 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.465056    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 29 21:45:40.465183 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.465099    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 29 21:45:40.465183 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.465123    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 29 21:45:40.465183 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.465150    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 29 21:45:40.465183 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.465165    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:45:40.465183 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.465170    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 29 21:45:40.465562 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.465200    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 29 21:45:40.465562 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.465205    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 29 21:45:40.465562 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.465251    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 29 21:45:40.465562 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.465268    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 29 21:45:40.465562 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.465332    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 29 21:45:40.465562 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.465343    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 29 21:45:40.465562 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.465333    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-6chnm\""
Oct 29 21:45:40.465562 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.465354    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ckcdt\""
Oct 29 21:45:40.465562 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.465350    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 29 21:45:40.465562 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.465379    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 29 21:45:40.465562 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.465379    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 29 21:45:40.465562 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.465390    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mlr8g\""
Oct 29 21:45:40.465963 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.465760    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7dqcg\""
Oct 29 21:45:40.470939 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.470918    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 29 21:45:40.471008 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.470964    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-znmmx\""
Oct 29 21:45:40.471008 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.470971    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 29 21:45:40.471008 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.470980    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 29 21:45:40.471107 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.471061    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6jh7q\""
Oct 29 21:45:40.503580 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.503563    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:45:40.511855 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:40.511834    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493b35d9c8008ff0c00d95f4c3e7e953.slice/crio-dfd9aba9499701d8616e24cc1e3b84b531f788c45a805468f92b6e2f9440ce3f WatchSource:0}: Error finding container dfd9aba9499701d8616e24cc1e3b84b531f788c45a805468f92b6e2f9440ce3f: Status 404 returned error can't find the container with id dfd9aba9499701d8616e24cc1e3b84b531f788c45a805468f92b6e2f9440ce3f
Oct 29 21:45:40.513056 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:40.513035    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db57e6b6465b77567369446b9e61568.slice/crio-1b7d7a6db961bfef862e90db2aca93842cf4863e6ab26c33af1b217ff4276fba WatchSource:0}: Error finding container 1b7d7a6db961bfef862e90db2aca93842cf4863e6ab26c33af1b217ff4276fba: Status 404 returned error can't find the container with id 1b7d7a6db961bfef862e90db2aca93842cf4863e6ab26c33af1b217ff4276fba
Oct 29 21:45:40.516230 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.516217    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 21:45:40.530229 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bhr9\" (UniqueName: \"kubernetes.io/projected/fc992647-89dd-44cc-ba19-3fdb03e125de-kube-api-access-9bhr9\") pod \"node-resolver-rrmxb\" (UID: \"fc992647-89dd-44cc-ba19-3fdb03e125de\") " pod="openshift-dns/node-resolver-rrmxb"
Oct 29 21:45:40.530315 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:45:40.530315 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530267    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-etc-selinux\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:45:40.530315 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530303    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-slash\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.530443 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-tuned\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.530443 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-etc-kubernetes\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.530443 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530352    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:45:40.530443 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530370    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfnx5\" (UniqueName: \"kubernetes.io/projected/8428734b-4a9f-4df9-ac06-81074198a9ba-kube-api-access-vfnx5\") pod \"node-ca-knxtf\" (UID: \"8428734b-4a9f-4df9-ac06-81074198a9ba\") " pod="openshift-image-registry/node-ca-knxtf"
Oct 29 21:45:40.530443 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af572bfb-940b-4226-b91f-d06554b40d3c-konnectivity-ca\") pod \"konnectivity-agent-g2rdv\" (UID: \"af572bfb-940b-4226-b91f-d06554b40d3c\") " pod="kube-system/konnectivity-agent-g2rdv"
Oct 29 21:45:40.530443 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530411    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-slash\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.530443 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530416    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5mps\" (UniqueName: \"kubernetes.io/projected/259e6b95-d64c-4101-871f-4a933d4ac38f-kube-api-access-v5mps\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.530443 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530415    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-etc-selinux\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:45:40.530443 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-sysctl-d\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.530850 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-sysctl-conf\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.530850 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530435    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-etc-kubernetes\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.530850 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530489    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbk9p\" (UniqueName: \"kubernetes.io/projected/9400c4a1-ccf5-4b45-94f8-e37079762e4f-kube-api-access-pbk9p\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:45:40.530850 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530515    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-registration-dir\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:45:40.530850 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530537    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/676904dc-4d4f-4bd6-b380-357d5d9a29d4-dbus\") pod \"global-pull-secret-syncer-hwq7z\" (UID: \"676904dc-4d4f-4bd6-b380-357d5d9a29d4\") " pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:45:40.530850 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brg9z\" (UniqueName: \"kubernetes.io/projected/f934ff4f-dadc-469b-a87e-16db95e3181f-kube-api-access-brg9z\") pod \"network-check-target-pkmzk\" (UID: \"f934ff4f-dadc-469b-a87e-16db95e3181f\") " pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:45:40.530850 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngw2c\" (UniqueName: \"kubernetes.io/projected/0febea83-955d-4a56-8bc4-c883a4141109-kube-api-access-ngw2c\") pod \"network-metrics-daemon-l7tkk\" (UID: \"0febea83-955d-4a56-8bc4-c883a4141109\") " pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:45:40.530850 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530606    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-sys\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.530850 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530609    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-sysctl-d\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.530850 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530628    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-registration-dir\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:45:40.530850 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-system-cni-dir\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.530850 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530681    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-cni-binary-copy\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.530850 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530693    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-system-cni-dir\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.530850 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530717    2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 29 21:45:40.530850 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530798    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-sysctl-conf\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.530850 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530832    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/676904dc-4d4f-4bd6-b380-357d5d9a29d4-dbus\") pod \"global-pull-secret-syncer-hwq7z\" (UID: \"676904dc-4d4f-4bd6-b380-357d5d9a29d4\") " pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:45:40.531617 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530877    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-sys\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.531617 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530888    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-multus-socket-dir-parent\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.531617 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-hostroot\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.531617 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530948    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-multus-conf-dir\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.531617 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1953d5bb-b35f-4c91-a263-a3664cf7fab2-iptables-alerter-script\") pod \"iptables-alerter-t997r\" (UID: \"1953d5bb-b35f-4c91-a263-a3664cf7fab2\") " pod="openshift-network-operator/iptables-alerter-t997r"
Oct 29 21:45:40.531617 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530976    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-multus-socket-dir-parent\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.531617 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.530996    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-run-netns\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.531617 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531000    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af572bfb-940b-4226-b91f-d06554b40d3c-konnectivity-ca\") pod \"konnectivity-agent-g2rdv\" (UID: \"af572bfb-940b-4226-b91f-d06554b40d3c\") " pod="kube-system/konnectivity-agent-g2rdv"
Oct 29 21:45:40.531617 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531019    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-kubernetes\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.531617 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531020    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-hostroot\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.531617 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531032    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-multus-conf-dir\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.531617 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531043    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9400c4a1-ccf5-4b45-94f8-e37079762e4f-system-cni-dir\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:45:40.531617 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-run-netns\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.531617 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1953d5bb-b35f-4c91-a263-a3664cf7fab2-host-slash\") pod \"iptables-alerter-t997r\" (UID: \"1953d5bb-b35f-4c91-a263-a3664cf7fab2\") " pod="openshift-network-operator/iptables-alerter-t997r"
Oct 29 21:45:40.531617 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531091    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-kubernetes\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.531617 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531097    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-run-netns\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.531617 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531109    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-socket-dir\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:45:40.531617 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-sys-fs\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:45:40.532462 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531133    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-run-netns\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.532462 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531148    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1953d5bb-b35f-4c91-a263-a3664cf7fab2-host-slash\") pod \"iptables-alerter-t997r\" (UID: \"1953d5bb-b35f-4c91-a263-a3664cf7fab2\") " pod="openshift-network-operator/iptables-alerter-t997r"
Oct 29 21:45:40.532462 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531170    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9400c4a1-ccf5-4b45-94f8-e37079762e4f-system-cni-dir\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:45:40.532462 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9400c4a1-ccf5-4b45-94f8-e37079762e4f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:45:40.532462 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9400c4a1-ccf5-4b45-94f8-e37079762e4f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:45:40.532462 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531221    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-sys-fs\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:45:40.532462 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc992647-89dd-44cc-ba19-3fdb03e125de-tmp-dir\") pod \"node-resolver-rrmxb\" (UID: \"fc992647-89dd-44cc-ba19-3fdb03e125de\") " pod="openshift-dns/node-resolver-rrmxb"
Oct 29 21:45:40.532462 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs\") pod \"network-metrics-daemon-l7tkk\" (UID: \"0febea83-955d-4a56-8bc4-c883a4141109\") " pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:45:40.532462 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-run-k8s-cni-cncf-io\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.532462 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-systemd-units\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.532462 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531351    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-cni-binary-copy\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.532462 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531369    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-socket-dir\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:45:40.532462 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531371    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-run-ovn\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.532462 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531408    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/259e6b95-d64c-4101-871f-4a933d4ac38f-ovnkube-config\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.532462 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531412    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-run-ovn\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.532462 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:40.531454    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:40.532462 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531434    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-lib-modules\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.532462 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531485    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-var-lib-openvswitch\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.533386 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-etc-openvswitch\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.533386 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531521    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-modprobe-d\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.533386 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531536    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x75hw\" (UniqueName: \"kubernetes.io/projected/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-kube-api-access-x75hw\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.533386 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531551    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8428734b-4a9f-4df9-ac06-81074198a9ba-serviceca\") pod \"node-ca-knxtf\" (UID: \"8428734b-4a9f-4df9-ac06-81074198a9ba\") " pod="openshift-image-registry/node-ca-knxtf"
Oct 29 21:45:40.533386 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531560    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9400c4a1-ccf5-4b45-94f8-e37079762e4f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:45:40.533386 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531567    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.533386 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531580    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1953d5bb-b35f-4c91-a263-a3664cf7fab2-iptables-alerter-script\") pod \"iptables-alerter-t997r\" (UID: \"1953d5bb-b35f-4c91-a263-a3664cf7fab2\") " pod="openshift-network-operator/iptables-alerter-t997r"
Oct 29 21:45:40.533386 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531596    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-sysconfig\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.533386 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531599    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.533386 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531632    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-etc-openvswitch\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.533386 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531643    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-sysconfig\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.533386 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531645    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-var-lib-cni-multus\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.533386 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531666    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-var-lib-cni-multus\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.533386 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531677    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-var-lib-openvswitch\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.533386 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531678    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af572bfb-940b-4226-b91f-d06554b40d3c-agent-certs\") pod \"konnectivity-agent-g2rdv\" (UID: \"af572bfb-940b-4226-b91f-d06554b40d3c\") " pod="kube-system/konnectivity-agent-g2rdv"
Oct 29 21:45:40.533386 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-device-dir\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:45:40.533386 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531739    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-lib-modules\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.534245 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531751    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc992647-89dd-44cc-ba19-3fdb03e125de-tmp-dir\") pod \"node-resolver-rrmxb\" (UID: \"fc992647-89dd-44cc-ba19-3fdb03e125de\") " pod="openshift-dns/node-resolver-rrmxb"
Oct 29 21:45:40.534245 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531737    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/676904dc-4d4f-4bd6-b380-357d5d9a29d4-original-pull-secret\") pod \"global-pull-secret-syncer-hwq7z\" (UID: \"676904dc-4d4f-4bd6-b380-357d5d9a29d4\") " pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:45:40.534245 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:40.531802    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs podName:0febea83-955d-4a56-8bc4-c883a4141109 nodeName:}" failed. No retries permitted until 2026-10-29 21:45:41.031752143 +0000 UTC m=+2.125381031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs") pod "network-metrics-daemon-l7tkk" (UID: "0febea83-955d-4a56-8bc4-c883a4141109") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:40.534245 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531802    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9400c4a1-ccf5-4b45-94f8-e37079762e4f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:45:40.534245 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:40.531815    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:40.534245 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-node-log\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.534245 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531849    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-node-log\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.534245 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:40.531858    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/676904dc-4d4f-4bd6-b380-357d5d9a29d4-original-pull-secret podName:676904dc-4d4f-4bd6-b380-357d5d9a29d4 nodeName:}" failed. No retries permitted until 2026-10-29 21:45:41.031843066 +0000 UTC m=+2.125471966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/676904dc-4d4f-4bd6-b380-357d5d9a29d4-original-pull-secret") pod "global-pull-secret-syncer-hwq7z" (UID: "676904dc-4d4f-4bd6-b380-357d5d9a29d4") : object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:40.534245 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531868    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-device-dir\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:45:40.534245 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531894    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-run-k8s-cni-cncf-io\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.534245 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531897    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-run-ovn-kubernetes\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.534245 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531926    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-run\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.534245 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531950    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7dvt\" (UniqueName: \"kubernetes.io/projected/b546ae66-9568-4840-a4e5-659d93eb80c5-kube-api-access-x7dvt\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.534245 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-run-multus-certs\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.534245 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.531999    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-run-ovn-kubernetes\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.534245 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/259e6b95-d64c-4101-871f-4a933d4ac38f-ovn-node-metrics-cert\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.534980 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532029    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-systemd-units\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.534980 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532039    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-var-lib-cni-bin\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.534980 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-var-lib-kubelet\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.534980 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532090    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62lfn\" (UniqueName: \"kubernetes.io/projected/ce551e36-0249-4723-b814-a98f0a282eed-kube-api-access-62lfn\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:45:40.534980 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-kubelet\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.534980 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-run-openvswitch\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.534980 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532158    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9400c4a1-ccf5-4b45-94f8-e37079762e4f-cnibin\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:45:40.534980 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-multus-daemon-config\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.534980 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8428734b-4a9f-4df9-ac06-81074198a9ba-host\") pod \"node-ca-knxtf\" (UID: \"8428734b-4a9f-4df9-ac06-81074198a9ba\") " pod="openshift-image-registry/node-ca-knxtf"
Oct 29 21:45:40.534980 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kr4j\" (UniqueName: \"kubernetes.io/projected/1953d5bb-b35f-4c91-a263-a3664cf7fab2-kube-api-access-2kr4j\") pod \"iptables-alerter-t997r\" (UID: \"1953d5bb-b35f-4c91-a263-a3664cf7fab2\") " pod="openshift-network-operator/iptables-alerter-t997r"
Oct 29 21:45:40.534980 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532246    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8428734b-4a9f-4df9-ac06-81074198a9ba-serviceca\") pod \"node-ca-knxtf\" (UID: \"8428734b-4a9f-4df9-ac06-81074198a9ba\") " pod="openshift-image-registry/node-ca-knxtf"
Oct 29 21:45:40.534980 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532253    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/676904dc-4d4f-4bd6-b380-357d5d9a29d4-kubelet-config\") pod \"global-pull-secret-syncer-hwq7z\" (UID: \"676904dc-4d4f-4bd6-b380-357d5d9a29d4\") " pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:45:40.534980 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532262    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-run\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.534980 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-log-socket\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.534980 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/259e6b95-d64c-4101-871f-4a933d4ac38f-env-overrides\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.534980 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532331    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-run-openvswitch\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.534980 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532343    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b546ae66-9568-4840-a4e5-659d93eb80c5-tmp\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.534980 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532356    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-modprobe-d\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.535477 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532364    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-multus-cni-dir\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.535477 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532373    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-var-lib-cni-bin\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.535477 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532386    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-cni-bin\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.535477 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-cni-netd\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.535477 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532411    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-var-lib-kubelet\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.535477 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532419    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9400c4a1-ccf5-4b45-94f8-e37079762e4f-cnibin\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:45:40.535477 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/259e6b95-d64c-4101-871f-4a933d4ac38f-ovnkube-script-lib\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.535477 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532468    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8428734b-4a9f-4df9-ac06-81074198a9ba-host\") pod \"node-ca-knxtf\" (UID: \"8428734b-4a9f-4df9-ac06-81074198a9ba\") " pod="openshift-image-registry/node-ca-knxtf"
Oct 29 21:45:40.535477 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-var-lib-kubelet\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.535477 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532533    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-log-socket\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.535477 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-host\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.535477 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532569    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9400c4a1-ccf5-4b45-94f8-e37079762e4f-cni-binary-copy\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:45:40.535477 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532580    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-host\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.535477 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc992647-89dd-44cc-ba19-3fdb03e125de-hosts-file\") pod \"node-resolver-rrmxb\" (UID: \"fc992647-89dd-44cc-ba19-3fdb03e125de\") " pod="openshift-dns/node-resolver-rrmxb"
Oct 29 21:45:40.535477 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532619    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-var-lib-kubelet\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.535477 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532620    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-cni-bin\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.535477 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532494    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/676904dc-4d4f-4bd6-b380-357d5d9a29d4-kubelet-config\") pod \"global-pull-secret-syncer-hwq7z\" (UID: \"676904dc-4d4f-4bd6-b380-357d5d9a29d4\") " pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:45:40.535477 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532405    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/259e6b95-d64c-4101-871f-4a933d4ac38f-ovnkube-config\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.535939 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532670    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-run-multus-certs\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.535939 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532677    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-run-systemd\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.535939 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532696    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-kubelet\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.535939 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532746    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-multus-cni-dir\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.535939 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532622    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-run-systemd\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.535939 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532789    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-systemd\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.535939 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532812    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9400c4a1-ccf5-4b45-94f8-e37079762e4f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:45:40.535939 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532840    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-cnibin\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.535939 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532856    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-multus-daemon-config\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.535939 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532913    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-cnibin\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.535939 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532948    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc992647-89dd-44cc-ba19-3fdb03e125de-hosts-file\") pod \"node-resolver-rrmxb\" (UID: \"fc992647-89dd-44cc-ba19-3fdb03e125de\") " pod="openshift-dns/node-resolver-rrmxb"
Oct 29 21:45:40.535939 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.532955    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-cni-netd\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.535939 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.533005    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-systemd\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.535939 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.533926    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9400c4a1-ccf5-4b45-94f8-e37079762e4f-cni-binary-copy\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:45:40.535939 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.534198    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9400c4a1-ccf5-4b45-94f8-e37079762e4f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:45:40.535939 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.534494    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/259e6b95-d64c-4101-871f-4a933d4ac38f-env-overrides\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.535939 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.534502    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-tuned\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.535939 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.534553    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/259e6b95-d64c-4101-871f-4a933d4ac38f-ovnkube-script-lib\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.536450 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.534813    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/259e6b95-d64c-4101-871f-4a933d4ac38f-ovn-node-metrics-cert\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.536450 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.534824    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af572bfb-940b-4226-b91f-d06554b40d3c-agent-certs\") pod \"konnectivity-agent-g2rdv\" (UID: \"af572bfb-940b-4226-b91f-d06554b40d3c\") " pod="kube-system/konnectivity-agent-g2rdv"
Oct 29 21:45:40.536450 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.534971    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b546ae66-9568-4840-a4e5-659d93eb80c5-tmp\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.555319 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.555281    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfnx5\" (UniqueName: \"kubernetes.io/projected/8428734b-4a9f-4df9-ac06-81074198a9ba-kube-api-access-vfnx5\") pod \"node-ca-knxtf\" (UID: \"8428734b-4a9f-4df9-ac06-81074198a9ba\") " pod="openshift-image-registry/node-ca-knxtf"
Oct 29 21:45:40.555461 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.555388    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbk9p\" (UniqueName: \"kubernetes.io/projected/9400c4a1-ccf5-4b45-94f8-e37079762e4f-kube-api-access-pbk9p\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:45:40.555555 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.555475    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngw2c\" (UniqueName: \"kubernetes.io/projected/0febea83-955d-4a56-8bc4-c883a4141109-kube-api-access-ngw2c\") pod \"network-metrics-daemon-l7tkk\" (UID: \"0febea83-955d-4a56-8bc4-c883a4141109\") " pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:45:40.555555 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:40.555537    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 21:45:40.555555 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:40.555556    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 21:45:40.555716 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:40.555571    2579 projected.go:196] Error preparing data for projected volume kube-api-access-brg9z for pod openshift-network-diagnostics/network-check-target-pkmzk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:40.555716 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:40.555631    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f934ff4f-dadc-469b-a87e-16db95e3181f-kube-api-access-brg9z podName:f934ff4f-dadc-469b-a87e-16db95e3181f nodeName:}" failed. No retries permitted until 2026-10-29 21:45:41.055615573 +0000 UTC m=+2.149244482 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-brg9z" (UniqueName: "kubernetes.io/projected/f934ff4f-dadc-469b-a87e-16db95e3181f-kube-api-access-brg9z") pod "network-check-target-pkmzk" (UID: "f934ff4f-dadc-469b-a87e-16db95e3181f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:40.555856 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.555793    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x75hw\" (UniqueName: \"kubernetes.io/projected/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-kube-api-access-x75hw\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.556644 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.556621    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5mps\" (UniqueName: \"kubernetes.io/projected/259e6b95-d64c-4101-871f-4a933d4ac38f-kube-api-access-v5mps\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.556644 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.556636    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bhr9\" (UniqueName: \"kubernetes.io/projected/fc992647-89dd-44cc-ba19-3fdb03e125de-kube-api-access-9bhr9\") pod \"node-resolver-rrmxb\" (UID: \"fc992647-89dd-44cc-ba19-3fdb03e125de\") " pod="openshift-dns/node-resolver-rrmxb"
Oct 29 21:45:40.557406 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.557387    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kr4j\" (UniqueName: \"kubernetes.io/projected/1953d5bb-b35f-4c91-a263-a3664cf7fab2-kube-api-access-2kr4j\") pod \"iptables-alerter-t997r\" (UID: \"1953d5bb-b35f-4c91-a263-a3664cf7fab2\") " pod="openshift-network-operator/iptables-alerter-t997r"
Oct 29 21:45:40.557552 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.557535    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62lfn\" (UniqueName: \"kubernetes.io/projected/ce551e36-0249-4723-b814-a98f0a282eed-kube-api-access-62lfn\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:45:40.557946 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.557931    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7dvt\" (UniqueName: \"kubernetes.io/projected/b546ae66-9568-4840-a4e5-659d93eb80c5-kube-api-access-x7dvt\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.622442 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.622401    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:45:40.684328 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.684306    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t997r"
Oct 29 21:45:40.686608 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.686590    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-g2rdv"
Oct 29 21:45:40.694408 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:40.694389    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1953d5bb_b35f_4c91_a263_a3664cf7fab2.slice/crio-f99ebf26277a5267b307753c364de36f94b220110643560650df779472db037f WatchSource:0}: Error finding container f99ebf26277a5267b307753c364de36f94b220110643560650df779472db037f: Status 404 returned error can't find the container with id f99ebf26277a5267b307753c364de36f94b220110643560650df779472db037f
Oct 29 21:45:40.696341 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:40.696319    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf572bfb_940b_4226_b91f_d06554b40d3c.slice/crio-6be2381599337d6580cefd43f997ec6eff8e7b9f84eefb7e4b1fa83a5c2be5e9 WatchSource:0}: Error finding container 6be2381599337d6580cefd43f997ec6eff8e7b9f84eefb7e4b1fa83a5c2be5e9: Status 404 returned error can't find the container with id 6be2381599337d6580cefd43f997ec6eff8e7b9f84eefb7e4b1fa83a5c2be5e9
Oct 29 21:45:40.720413 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.720393    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:45:40.729254 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:40.729231    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce551e36_0249_4723_b814_a98f0a282eed.slice/crio-e2f05788772608ca6575554c6aa26b79b2e8da04675dbbcc33d6bd2e67eb5292 WatchSource:0}: Error finding container e2f05788772608ca6575554c6aa26b79b2e8da04675dbbcc33d6bd2e67eb5292: Status 404 returned error can't find the container with id e2f05788772608ca6575554c6aa26b79b2e8da04675dbbcc33d6bd2e67eb5292
Oct 29 21:45:40.738069 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.738051    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:45:40.740146 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.740130    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:45:40.748386 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:40.748357    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259e6b95_d64c_4101_871f_4a933d4ac38f.slice/crio-6432ecd21ecbbc981dc032e54f0a8426520ef280116b9d595649f14ac25a7f07 WatchSource:0}: Error finding container 6432ecd21ecbbc981dc032e54f0a8426520ef280116b9d595649f14ac25a7f07: Status 404 returned error can't find the container with id 6432ecd21ecbbc981dc032e54f0a8426520ef280116b9d595649f14ac25a7f07
Oct 29 21:45:40.748885 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:40.748831    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb546ae66_9568_4840_a4e5_659d93eb80c5.slice/crio-11d97da879728c462bff4845cbec74c2a1fd055544dca3bd08b49e45472bf28d WatchSource:0}: Error finding container 11d97da879728c462bff4845cbec74c2a1fd055544dca3bd08b49e45472bf28d: Status 404 returned error can't find the container with id 11d97da879728c462bff4845cbec74c2a1fd055544dca3bd08b49e45472bf28d
Oct 29 21:45:40.756659 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.756643    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:45:40.765482 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:40.765462    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9400c4a1_ccf5_4b45_94f8_e37079762e4f.slice/crio-cc8c68976e0a46c1f637ff27ab50995b2c2befc957d10003c1bebef8c4c9a8a6 WatchSource:0}: Error finding container cc8c68976e0a46c1f637ff27ab50995b2c2befc957d10003c1bebef8c4c9a8a6: Status 404 returned error can't find the container with id cc8c68976e0a46c1f637ff27ab50995b2c2befc957d10003c1bebef8c4c9a8a6
Oct 29 21:45:40.768381 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.768356    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ts2kk"
Oct 29 21:45:40.773915 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.773902    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rrmxb"
Oct 29 21:45:40.776297 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:40.776262    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae48f1b_bbf0_4e82_8eeb_33f9c1f27814.slice/crio-95ecdf2bd50b82bfda86de3b56e6c5a5f58481b3f4f64f69fd1abc173fbda723 WatchSource:0}: Error finding container 95ecdf2bd50b82bfda86de3b56e6c5a5f58481b3f4f64f69fd1abc173fbda723: Status 404 returned error can't find the container with id 95ecdf2bd50b82bfda86de3b56e6c5a5f58481b3f4f64f69fd1abc173fbda723
Oct 29 21:45:40.777325 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:40.777305    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-knxtf"
Oct 29 21:45:40.781660 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:40.781640    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc992647_89dd_44cc_ba19_3fdb03e125de.slice/crio-29f62b2ea7423ce44a08ae94c6c695b2e93bfbdb81d23930cffaab45c8db081c WatchSource:0}: Error finding container 29f62b2ea7423ce44a08ae94c6c695b2e93bfbdb81d23930cffaab45c8db081c: Status 404 returned error can't find the container with id 29f62b2ea7423ce44a08ae94c6c695b2e93bfbdb81d23930cffaab45c8db081c
Oct 29 21:45:40.786048 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:45:40.786028    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8428734b_4a9f_4df9_ac06_81074198a9ba.slice/crio-8a83146c0d573217f00980b7fd41c8c1b474aa36a050d75b280b82c8ece5681b WatchSource:0}: Error finding container 8a83146c0d573217f00980b7fd41c8c1b474aa36a050d75b280b82c8ece5681b: Status 404 returned error can't find the container with id 8a83146c0d573217f00980b7fd41c8c1b474aa36a050d75b280b82c8ece5681b
Oct 29 21:45:41.036331 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:41.036241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs\") pod \"network-metrics-daemon-l7tkk\" (UID: \"0febea83-955d-4a56-8bc4-c883a4141109\") " pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:45:41.036331 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:41.036307    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/676904dc-4d4f-4bd6-b380-357d5d9a29d4-original-pull-secret\") pod \"global-pull-secret-syncer-hwq7z\" (UID: \"676904dc-4d4f-4bd6-b380-357d5d9a29d4\") " pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:45:41.036527 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:41.036425    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:41.036527 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:41.036480    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/676904dc-4d4f-4bd6-b380-357d5d9a29d4-original-pull-secret podName:676904dc-4d4f-4bd6-b380-357d5d9a29d4 nodeName:}" failed. No retries permitted until 2026-10-29 21:45:42.03646234 +0000 UTC m=+3.130091245 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/676904dc-4d4f-4bd6-b380-357d5d9a29d4-original-pull-secret") pod "global-pull-secret-syncer-hwq7z" (UID: "676904dc-4d4f-4bd6-b380-357d5d9a29d4") : object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:41.036639 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:41.036550    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:41.036639 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:41.036588    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs podName:0febea83-955d-4a56-8bc4-c883a4141109 nodeName:}" failed. No retries permitted until 2026-10-29 21:45:42.03657602 +0000 UTC m=+3.130204908 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs") pod "network-metrics-daemon-l7tkk" (UID: "0febea83-955d-4a56-8bc4-c883a4141109") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:41.137388 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:41.137355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brg9z\" (UniqueName: \"kubernetes.io/projected/f934ff4f-dadc-469b-a87e-16db95e3181f-kube-api-access-brg9z\") pod \"network-check-target-pkmzk\" (UID: \"f934ff4f-dadc-469b-a87e-16db95e3181f\") " pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:45:41.137574 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:41.137559    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 21:45:41.137637 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:41.137582    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 21:45:41.137637 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:41.137594    2579 projected.go:196] Error preparing data for projected volume kube-api-access-brg9z for pod openshift-network-diagnostics/network-check-target-pkmzk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:41.137746 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:41.137646    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f934ff4f-dadc-469b-a87e-16db95e3181f-kube-api-access-brg9z podName:f934ff4f-dadc-469b-a87e-16db95e3181f nodeName:}" failed. No retries permitted until 2026-10-29 21:45:42.137629741 +0000 UTC m=+3.231258644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-brg9z" (UniqueName: "kubernetes.io/projected/f934ff4f-dadc-469b-a87e-16db95e3181f-kube-api-access-brg9z") pod "network-check-target-pkmzk" (UID: "f934ff4f-dadc-469b-a87e-16db95e3181f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:41.417353 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:41.417232    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 21:40:40 +0000 UTC" deadline="2028-08-07 19:54:40.92999998 +0000 UTC"
Oct 29 21:45:41.417353 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:41.417297    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15550h8m59.512721389s"
Oct 29 21:45:41.511733 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:41.511704    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:45:41.511875 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:41.511832    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7tkk" podUID="0febea83-955d-4a56-8bc4-c883a4141109"
Oct 29 21:45:41.526410 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:41.526328    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ts2kk" event={"ID":"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814","Type":"ContainerStarted","Data":"95ecdf2bd50b82bfda86de3b56e6c5a5f58481b3f4f64f69fd1abc173fbda723"}
Oct 29 21:45:41.528239 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:41.528205    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzjtw" event={"ID":"9400c4a1-ccf5-4b45-94f8-e37079762e4f","Type":"ContainerStarted","Data":"cc8c68976e0a46c1f637ff27ab50995b2c2befc957d10003c1bebef8c4c9a8a6"}
Oct 29 21:45:41.531636 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:41.531612    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74bdq" event={"ID":"259e6b95-d64c-4101-871f-4a933d4ac38f","Type":"ContainerStarted","Data":"6432ecd21ecbbc981dc032e54f0a8426520ef280116b9d595649f14ac25a7f07"}
Oct 29 21:45:41.533841 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:41.533817    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal" event={"ID":"493b35d9c8008ff0c00d95f4c3e7e953","Type":"ContainerStarted","Data":"dfd9aba9499701d8616e24cc1e3b84b531f788c45a805468f92b6e2f9440ce3f"}
Oct 29 21:45:41.537953 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:41.537928    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-knxtf" event={"ID":"8428734b-4a9f-4df9-ac06-81074198a9ba","Type":"ContainerStarted","Data":"8a83146c0d573217f00980b7fd41c8c1b474aa36a050d75b280b82c8ece5681b"}
Oct 29 21:45:41.542830 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:41.542766    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rrmxb" event={"ID":"fc992647-89dd-44cc-ba19-3fdb03e125de","Type":"ContainerStarted","Data":"29f62b2ea7423ce44a08ae94c6c695b2e93bfbdb81d23930cffaab45c8db081c"}
Oct 29 21:45:41.545239 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:41.545216    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s6mfd" event={"ID":"b546ae66-9568-4840-a4e5-659d93eb80c5","Type":"ContainerStarted","Data":"11d97da879728c462bff4845cbec74c2a1fd055544dca3bd08b49e45472bf28d"}
Oct 29 21:45:41.549980 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:41.549942    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz" event={"ID":"ce551e36-0249-4723-b814-a98f0a282eed","Type":"ContainerStarted","Data":"e2f05788772608ca6575554c6aa26b79b2e8da04675dbbcc33d6bd2e67eb5292"}
Oct 29 21:45:41.558493 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:41.558472    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g2rdv" event={"ID":"af572bfb-940b-4226-b91f-d06554b40d3c","Type":"ContainerStarted","Data":"6be2381599337d6580cefd43f997ec6eff8e7b9f84eefb7e4b1fa83a5c2be5e9"}
Oct 29 21:45:41.566576 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:41.566546    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t997r" event={"ID":"1953d5bb-b35f-4c91-a263-a3664cf7fab2","Type":"ContainerStarted","Data":"f99ebf26277a5267b307753c364de36f94b220110643560650df779472db037f"}
Oct 29 21:45:41.568524 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:41.568462    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal" event={"ID":"4db57e6b6465b77567369446b9e61568","Type":"ContainerStarted","Data":"1b7d7a6db961bfef862e90db2aca93842cf4863e6ab26c33af1b217ff4276fba"}
Oct 29 21:45:41.774779 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:41.774708    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:45:42.045810 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:42.045012    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs\") pod \"network-metrics-daemon-l7tkk\" (UID: \"0febea83-955d-4a56-8bc4-c883a4141109\") " pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:45:42.045810 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:42.045063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/676904dc-4d4f-4bd6-b380-357d5d9a29d4-original-pull-secret\") pod \"global-pull-secret-syncer-hwq7z\" (UID: \"676904dc-4d4f-4bd6-b380-357d5d9a29d4\") " pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:45:42.045810 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:42.045184    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:42.045810 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:42.045245    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/676904dc-4d4f-4bd6-b380-357d5d9a29d4-original-pull-secret podName:676904dc-4d4f-4bd6-b380-357d5d9a29d4 nodeName:}" failed. No retries permitted until 2026-10-29 21:45:44.045227633 +0000 UTC m=+5.138856526 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/676904dc-4d4f-4bd6-b380-357d5d9a29d4-original-pull-secret") pod "global-pull-secret-syncer-hwq7z" (UID: "676904dc-4d4f-4bd6-b380-357d5d9a29d4") : object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:42.045810 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:42.045643    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:42.045810 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:42.045692    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs podName:0febea83-955d-4a56-8bc4-c883a4141109 nodeName:}" failed. No retries permitted until 2026-10-29 21:45:44.045675288 +0000 UTC m=+5.139304196 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs") pod "network-metrics-daemon-l7tkk" (UID: "0febea83-955d-4a56-8bc4-c883a4141109") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:42.145728 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:42.145674    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brg9z\" (UniqueName: \"kubernetes.io/projected/f934ff4f-dadc-469b-a87e-16db95e3181f-kube-api-access-brg9z\") pod \"network-check-target-pkmzk\" (UID: \"f934ff4f-dadc-469b-a87e-16db95e3181f\") " pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:45:42.146040 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:42.145927    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 21:45:42.146040 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:42.145954    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 21:45:42.146040 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:42.145967    2579 projected.go:196] Error preparing data for projected volume kube-api-access-brg9z for pod openshift-network-diagnostics/network-check-target-pkmzk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:42.146040 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:42.146020    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f934ff4f-dadc-469b-a87e-16db95e3181f-kube-api-access-brg9z podName:f934ff4f-dadc-469b-a87e-16db95e3181f nodeName:}" failed. No retries permitted until 2026-10-29 21:45:44.146002208 +0000 UTC m=+5.239631108 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-brg9z" (UniqueName: "kubernetes.io/projected/f934ff4f-dadc-469b-a87e-16db95e3181f-kube-api-access-brg9z") pod "network-check-target-pkmzk" (UID: "f934ff4f-dadc-469b-a87e-16db95e3181f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:42.164003 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:42.163978    2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:45:42.418124 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:42.418043    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 21:40:40 +0000 UTC" deadline="2028-05-12 07:44:35.200313368 +0000 UTC"
Oct 29 21:45:42.418124 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:42.418079    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13449h58m52.782238325s"
Oct 29 21:45:42.508736 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:42.508705    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:45:42.508903 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:42.508839    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkmzk" podUID="f934ff4f-dadc-469b-a87e-16db95e3181f"
Oct 29 21:45:42.509259 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:42.509242    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:45:42.509439 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:42.509415    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwq7z" podUID="676904dc-4d4f-4bd6-b380-357d5d9a29d4"
Oct 29 21:45:43.516297 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:43.516259    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:45:43.516670 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:43.516402    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwq7z" podUID="676904dc-4d4f-4bd6-b380-357d5d9a29d4"
Oct 29 21:45:43.516779 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:43.516759    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:45:43.516904 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:43.516883    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7tkk" podUID="0febea83-955d-4a56-8bc4-c883a4141109"
Oct 29 21:45:44.062794 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:44.062761    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs\") pod \"network-metrics-daemon-l7tkk\" (UID: \"0febea83-955d-4a56-8bc4-c883a4141109\") " pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:45:44.062956 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:44.062809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/676904dc-4d4f-4bd6-b380-357d5d9a29d4-original-pull-secret\") pod \"global-pull-secret-syncer-hwq7z\" (UID: \"676904dc-4d4f-4bd6-b380-357d5d9a29d4\") " pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:45:44.063015 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:44.062959    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:44.063066 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:44.063014    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/676904dc-4d4f-4bd6-b380-357d5d9a29d4-original-pull-secret podName:676904dc-4d4f-4bd6-b380-357d5d9a29d4 nodeName:}" failed. No retries permitted until 2026-10-29 21:45:48.062997733 +0000 UTC m=+9.156626622 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/676904dc-4d4f-4bd6-b380-357d5d9a29d4-original-pull-secret") pod "global-pull-secret-syncer-hwq7z" (UID: "676904dc-4d4f-4bd6-b380-357d5d9a29d4") : object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:44.063376 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:44.063356    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:44.063473 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:44.063404    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs podName:0febea83-955d-4a56-8bc4-c883a4141109 nodeName:}" failed. No retries permitted until 2026-10-29 21:45:48.063393559 +0000 UTC m=+9.157022457 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs") pod "network-metrics-daemon-l7tkk" (UID: "0febea83-955d-4a56-8bc4-c883a4141109") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:44.163580 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:44.163543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brg9z\" (UniqueName: \"kubernetes.io/projected/f934ff4f-dadc-469b-a87e-16db95e3181f-kube-api-access-brg9z\") pod \"network-check-target-pkmzk\" (UID: \"f934ff4f-dadc-469b-a87e-16db95e3181f\") " pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:45:44.163753 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:44.163737    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 21:45:44.163832 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:44.163759    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 21:45:44.163832 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:44.163769    2579 projected.go:196] Error preparing data for projected volume kube-api-access-brg9z for pod openshift-network-diagnostics/network-check-target-pkmzk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:44.163832 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:44.163825    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f934ff4f-dadc-469b-a87e-16db95e3181f-kube-api-access-brg9z podName:f934ff4f-dadc-469b-a87e-16db95e3181f nodeName:}" failed. No retries permitted until 2026-10-29 21:45:48.163807076 +0000 UTC m=+9.257435981 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-brg9z" (UniqueName: "kubernetes.io/projected/f934ff4f-dadc-469b-a87e-16db95e3181f-kube-api-access-brg9z") pod "network-check-target-pkmzk" (UID: "f934ff4f-dadc-469b-a87e-16db95e3181f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:44.509547 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:44.508505    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:45:44.509547 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:44.508776    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkmzk" podUID="f934ff4f-dadc-469b-a87e-16db95e3181f"
Oct 29 21:45:44.594194 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:44.594158    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzjtw" event={"ID":"9400c4a1-ccf5-4b45-94f8-e37079762e4f","Type":"ContainerStarted","Data":"6e9dc9d73f2e678b39692bcf2a191d2d79352e50ccd4d322849a8e3fc6fbef2b"}
Oct 29 21:45:44.616414 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:44.615796    2579 generic.go:356] "Generic (PLEG): container finished" podID="493b35d9c8008ff0c00d95f4c3e7e953" containerID="d40fb390702c1e1a031beead78275082701bf35228ebb6151209a529a6aa3ffc" exitCode=0
Oct 29 21:45:44.616414 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:44.615845    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal" event={"ID":"493b35d9c8008ff0c00d95f4c3e7e953","Type":"ContainerDied","Data":"d40fb390702c1e1a031beead78275082701bf35228ebb6151209a529a6aa3ffc"}
Oct 29 21:45:45.511799 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:45.511727    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:45:45.511948 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:45.511854    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwq7z" podUID="676904dc-4d4f-4bd6-b380-357d5d9a29d4"
Oct 29 21:45:45.512301 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:45.512268    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:45:45.512434 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:45.512411    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7tkk" podUID="0febea83-955d-4a56-8bc4-c883a4141109"
Oct 29 21:45:45.622484 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:45.622447    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal" event={"ID":"493b35d9c8008ff0c00d95f4c3e7e953","Type":"ContainerStarted","Data":"7b067d859195a198b899849ee55f0045da39c91e2aa9ab7b399e37e4a9e98d50"}
Oct 29 21:45:45.626982 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:45.626954    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-g2rdv" event={"ID":"af572bfb-940b-4226-b91f-d06554b40d3c","Type":"ContainerStarted","Data":"126b5c12bd569048cf718524ee0af52b9eb18b64560f4cec462520b06480c066"}
Oct 29 21:45:45.630819 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:45.630794    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal" event={"ID":"4db57e6b6465b77567369446b9e61568","Type":"ContainerStarted","Data":"5d1f40acaa9a7bef0f4497c34ad4ddd4cd7c646c42c9d1439579502bccb6c6d1"}
Oct 29 21:45:45.633189 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:45.633167    2579 generic.go:356] "Generic (PLEG): container finished" podID="9400c4a1-ccf5-4b45-94f8-e37079762e4f" containerID="6e9dc9d73f2e678b39692bcf2a191d2d79352e50ccd4d322849a8e3fc6fbef2b" exitCode=0
Oct 29 21:45:45.633300 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:45.633195    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzjtw" event={"ID":"9400c4a1-ccf5-4b45-94f8-e37079762e4f","Type":"ContainerDied","Data":"6e9dc9d73f2e678b39692bcf2a191d2d79352e50ccd4d322849a8e3fc6fbef2b"}
Oct 29 21:45:45.714801 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:45.714740    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal" podStartSLOduration=5.714722055 podStartE2EDuration="5.714722055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:45:45.667672717 +0000 UTC m=+6.761301625" watchObservedRunningTime="2026-10-29 21:45:45.714722055 +0000 UTC m=+6.808350963"
Oct 29 21:45:45.765996 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:45.765907    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal" podStartSLOduration=5.765889382 podStartE2EDuration="5.765889382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:45:45.715339495 +0000 UTC m=+6.808968406" watchObservedRunningTime="2026-10-29 21:45:45.765889382 +0000 UTC m=+6.859518289"
Oct 29 21:45:46.183960 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:46.183935    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-g2rdv"
Oct 29 21:45:46.184511 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:46.184493    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-g2rdv"
Oct 29 21:45:46.230467 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:46.230416    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-g2rdv" podStartSLOduration=2.780357467 podStartE2EDuration="7.230399764s" totalImagesPullingTime="4.450042297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:45:45.81081461 +0000 UTC m=+6.904443520" watchObservedRunningTime="2026-10-29 21:45:46.230399764 +0000 UTC m=+7.324028675"
Oct 29 21:45:46.509071 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:46.508999    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:45:46.509230 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:46.509126    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkmzk" podUID="f934ff4f-dadc-469b-a87e-16db95e3181f"
Oct 29 21:45:46.638639 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:46.638604    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-knxtf" event={"ID":"8428734b-4a9f-4df9-ac06-81074198a9ba","Type":"ContainerStarted","Data":"c7a34ea3aab40b73a218f1d9c277a2452aa899872338a68123a4f5524bc19968"}
Oct 29 21:45:47.511067 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:47.509455    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:45:47.511067 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:47.509499    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:45:47.511067 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:47.509566    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7tkk" podUID="0febea83-955d-4a56-8bc4-c883a4141109"
Oct 29 21:45:47.511067 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:47.509727    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwq7z" podUID="676904dc-4d4f-4bd6-b380-357d5d9a29d4"
Oct 29 21:45:47.646136 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:47.646105    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz" event={"ID":"ce551e36-0249-4723-b814-a98f0a282eed","Type":"ContainerStarted","Data":"b4c84b12c7da7f27616e6b8acd7238af28d2b166c87d0e93f9500c52cae4e105"}
Oct 29 21:45:47.646606 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:47.646183    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 21:45:48.099607 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:48.099571    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs\") pod \"network-metrics-daemon-l7tkk\" (UID: \"0febea83-955d-4a56-8bc4-c883a4141109\") " pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:45:48.099776 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:48.099632    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/676904dc-4d4f-4bd6-b380-357d5d9a29d4-original-pull-secret\") pod \"global-pull-secret-syncer-hwq7z\" (UID: \"676904dc-4d4f-4bd6-b380-357d5d9a29d4\") " pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:45:48.099776 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:48.099729    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:48.099776 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:48.099777    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/676904dc-4d4f-4bd6-b380-357d5d9a29d4-original-pull-secret podName:676904dc-4d4f-4bd6-b380-357d5d9a29d4 nodeName:}" failed. No retries permitted until 2026-10-29 21:45:56.099763308 +0000 UTC m=+17.193392194 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/676904dc-4d4f-4bd6-b380-357d5d9a29d4-original-pull-secret") pod "global-pull-secret-syncer-hwq7z" (UID: "676904dc-4d4f-4bd6-b380-357d5d9a29d4") : object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:48.099940 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:48.099820    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:48.099940 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:48.099841    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs podName:0febea83-955d-4a56-8bc4-c883a4141109 nodeName:}" failed. No retries permitted until 2026-10-29 21:45:56.099834088 +0000 UTC m=+17.193462974 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs") pod "network-metrics-daemon-l7tkk" (UID: "0febea83-955d-4a56-8bc4-c883a4141109") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:48.200880 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:48.200797    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brg9z\" (UniqueName: \"kubernetes.io/projected/f934ff4f-dadc-469b-a87e-16db95e3181f-kube-api-access-brg9z\") pod \"network-check-target-pkmzk\" (UID: \"f934ff4f-dadc-469b-a87e-16db95e3181f\") " pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:45:48.201025 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:48.200979    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 21:45:48.201025 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:48.201002    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 21:45:48.201025 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:48.201016    2579 projected.go:196] Error preparing data for projected volume kube-api-access-brg9z for pod openshift-network-diagnostics/network-check-target-pkmzk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:48.201185 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:48.201072    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f934ff4f-dadc-469b-a87e-16db95e3181f-kube-api-access-brg9z podName:f934ff4f-dadc-469b-a87e-16db95e3181f nodeName:}" failed. No retries permitted until 2026-10-29 21:45:56.201055185 +0000 UTC m=+17.294684084 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-brg9z" (UniqueName: "kubernetes.io/projected/f934ff4f-dadc-469b-a87e-16db95e3181f-kube-api-access-brg9z") pod "network-check-target-pkmzk" (UID: "f934ff4f-dadc-469b-a87e-16db95e3181f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:48.509963 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:48.509249    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:45:48.509963 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:48.509404    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkmzk" podUID="f934ff4f-dadc-469b-a87e-16db95e3181f"
Oct 29 21:45:48.709625 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:48.709603    2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 29 21:45:49.511905 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:49.511880    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:45:49.512011 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:49.511996    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwq7z" podUID="676904dc-4d4f-4bd6-b380-357d5d9a29d4"
Oct 29 21:45:49.512070 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:49.512053    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:45:49.512158 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:49.512126    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7tkk" podUID="0febea83-955d-4a56-8bc4-c883a4141109"
Oct 29 21:45:49.529669 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:49.529392    2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T21:45:48.709622734Z","UUID":"f0d88116-1884-4231-8cee-3a597d5710a3","Handler":null,"Name":"","Endpoint":""}
Oct 29 21:45:49.534760 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:49.534737    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 29 21:45:49.534849 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:49.534776    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 29 21:45:49.652013 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:49.651975    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rrmxb" event={"ID":"fc992647-89dd-44cc-ba19-3fdb03e125de","Type":"ContainerStarted","Data":"123ad49744cad6f72c9632e0b6b72ffaa8cf6dbae5797b1f3e430404c4a3098e"}
Oct 29 21:45:49.659894 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:49.655451    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz" event={"ID":"ce551e36-0249-4723-b814-a98f0a282eed","Type":"ContainerStarted","Data":"d55a372ee78798885a13c9fc421e0d323b8af12ff6a4577e45e42e7b4982bde5"}
Oct 29 21:45:49.659894 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:49.657549    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ts2kk" event={"ID":"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814","Type":"ContainerStarted","Data":"c98c2633958a94dcec58aaa5904219c3f06006fa89f2daa95bad9e5782a19f5d"}
Oct 29 21:45:49.659894 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:49.659501    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzjtw" event={"ID":"9400c4a1-ccf5-4b45-94f8-e37079762e4f","Type":"ContainerStarted","Data":"fc79f20e73ad002f644f41df19e1bb1f25667d0a086baf6a7f752d7523435cda"}
Oct 29 21:45:49.697079 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:49.697024    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-knxtf" podStartSLOduration=5.408491935 podStartE2EDuration="10.697006611s" totalImagesPullingTime="5.288514676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:45:46.683058817 +0000 UTC m=+7.776687726" watchObservedRunningTime="2026-10-29 21:45:49.697006611 +0000 UTC m=+10.790635521"
Oct 29 21:45:49.749303 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:49.749242    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-rrmxb" podStartSLOduration=2.643376704 podStartE2EDuration="10.749225812s" totalImagesPullingTime="8.105849108s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:45:49.697369302 +0000 UTC m=+10.790998212" watchObservedRunningTime="2026-10-29 21:45:49.749225812 +0000 UTC m=+10.842854721"
Oct 29 21:45:50.509207 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:50.509135    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:45:50.509384 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:50.509265    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkmzk" podUID="f934ff4f-dadc-469b-a87e-16db95e3181f"
Oct 29 21:45:50.665721 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:50.665686    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz" event={"ID":"ce551e36-0249-4723-b814-a98f0a282eed","Type":"ContainerStarted","Data":"8ddb48ac84bf1c01f8a8faab2e3c1a9c4f0addc66243bb212307ca3adc708bbb"}
Oct 29 21:45:50.668361 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:50.668331    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t997r" event={"ID":"1953d5bb-b35f-4c91-a263-a3664cf7fab2","Type":"ContainerStarted","Data":"d523e48a06585f6f9609d5d2cf4a71f94a137d7be67727bedf9f81a2a8a8c41a"}
Oct 29 21:45:50.670319 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:50.670277    2579 generic.go:356] "Generic (PLEG): container finished" podID="9400c4a1-ccf5-4b45-94f8-e37079762e4f" containerID="fc79f20e73ad002f644f41df19e1bb1f25667d0a086baf6a7f752d7523435cda" exitCode=0
Oct 29 21:45:50.670400 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:50.670349    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzjtw" event={"ID":"9400c4a1-ccf5-4b45-94f8-e37079762e4f","Type":"ContainerDied","Data":"fc79f20e73ad002f644f41df19e1bb1f25667d0a086baf6a7f752d7523435cda"}
Oct 29 21:45:50.714225 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:50.714172    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-ts2kk" podStartSLOduration=3.837177949 podStartE2EDuration="11.71415516s" totalImagesPullingTime="7.876977211s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:45:49.798009387 +0000 UTC m=+10.891638294" watchObservedRunningTime="2026-10-29 21:45:50.71415516 +0000 UTC m=+11.807784073"
Oct 29 21:45:50.714736 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:50.714706    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz" podStartSLOduration=3.140835139 podStartE2EDuration="11.714694895s" totalImagesPullingTime="8.573859756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:45:50.714005081 +0000 UTC m=+11.807633980" watchObservedRunningTime="2026-10-29 21:45:50.714694895 +0000 UTC m=+11.808323802"
Oct 29 21:45:50.819939 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:50.819851    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-t997r" podStartSLOduration=3.776297444 podStartE2EDuration="11.819837083s" totalImagesPullingTime="8.043539639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:45:50.760560875 +0000 UTC m=+11.854189784" watchObservedRunningTime="2026-10-29 21:45:50.819837083 +0000 UTC m=+11.913465992"
Oct 29 21:45:51.113949 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:51.112873    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-g2rdv"
Oct 29 21:45:51.113949 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:51.113006    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 21:45:51.114137 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:51.114039    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g2rdv"
Oct 29 21:45:51.508853 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:51.508784    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:45:51.508974 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:51.508930    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwq7z" podUID="676904dc-4d4f-4bd6-b380-357d5d9a29d4"
Oct 29 21:45:51.509023 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:51.508979    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:45:51.509119 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:51.509097    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7tkk" podUID="0febea83-955d-4a56-8bc4-c883a4141109"
Oct 29 21:45:51.673686 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:51.673650    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s6mfd" event={"ID":"b546ae66-9568-4840-a4e5-659d93eb80c5","Type":"ContainerStarted","Data":"5427db98b2c8075df327135f240c0bb1fdb7c55502996061dbad8db27e6b0483"}
Oct 29 21:45:51.676679 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:51.676652    2579 generic.go:356] "Generic (PLEG): container finished" podID="9400c4a1-ccf5-4b45-94f8-e37079762e4f" containerID="19ea69748f71f0d6efd89587b876593eee4fe7df45ca18bb8e082be12f5c1c1e" exitCode=0
Oct 29 21:45:51.676800 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:51.676749    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzjtw" event={"ID":"9400c4a1-ccf5-4b45-94f8-e37079762e4f","Type":"ContainerDied","Data":"19ea69748f71f0d6efd89587b876593eee4fe7df45ca18bb8e082be12f5c1c1e"}
Oct 29 21:45:51.720690 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:51.720647    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s6mfd" podStartSLOduration=2.542028619 podStartE2EDuration="12.72063572s" totalImagesPullingTime="10.178607101s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:45:51.720382685 +0000 UTC m=+12.814011595" watchObservedRunningTime="2026-10-29 21:45:51.72063572 +0000 UTC m=+12.814264628"
Oct 29 21:45:52.509064 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:52.509039    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:45:52.509706 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:52.509151    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkmzk" podUID="f934ff4f-dadc-469b-a87e-16db95e3181f"
Oct 29 21:45:52.682246 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:52.681987    2579 generic.go:356] "Generic (PLEG): container finished" podID="9400c4a1-ccf5-4b45-94f8-e37079762e4f" containerID="3cc736966201b4831f6ae362e62bf7742d447e94ec442532bb26c1b35fd0559f" exitCode=0
Oct 29 21:45:52.682246 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:52.682034    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzjtw" event={"ID":"9400c4a1-ccf5-4b45-94f8-e37079762e4f","Type":"ContainerDied","Data":"3cc736966201b4831f6ae362e62bf7742d447e94ec442532bb26c1b35fd0559f"}
Oct 29 21:45:53.509313 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:53.508948    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:45:53.509313 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:53.509063    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwq7z" podUID="676904dc-4d4f-4bd6-b380-357d5d9a29d4"
Oct 29 21:45:53.509710 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:53.509447    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:45:53.509710 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:53.509553    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7tkk" podUID="0febea83-955d-4a56-8bc4-c883a4141109"
Oct 29 21:45:54.509169 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:54.509136    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:45:54.509269 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:54.509248    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkmzk" podUID="f934ff4f-dadc-469b-a87e-16db95e3181f"
Oct 29 21:45:54.690034 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:54.689951    2579 generic.go:356] "Generic (PLEG): container finished" podID="9400c4a1-ccf5-4b45-94f8-e37079762e4f" containerID="ac7376cc4ea396e59e08092e2d7e19493bf127bdde5432ef1193d35b7e13c657" exitCode=0
Oct 29 21:45:54.690480 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:54.690030    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzjtw" event={"ID":"9400c4a1-ccf5-4b45-94f8-e37079762e4f","Type":"ContainerDied","Data":"ac7376cc4ea396e59e08092e2d7e19493bf127bdde5432ef1193d35b7e13c657"}
Oct 29 21:45:55.508930 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:55.508911    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:45:55.509020 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:55.509006    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwq7z" podUID="676904dc-4d4f-4bd6-b380-357d5d9a29d4"
Oct 29 21:45:55.509055 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:55.509044    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:45:55.509120 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:55.509097    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7tkk" podUID="0febea83-955d-4a56-8bc4-c883a4141109"
Oct 29 21:45:55.695492 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:55.695316    2579 generic.go:356] "Generic (PLEG): container finished" podID="9400c4a1-ccf5-4b45-94f8-e37079762e4f" containerID="6f3433933e446ad2be68d92dd40adf255d899677bb24c1a56cb2e75de77353ae" exitCode=0
Oct 29 21:45:55.695883 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:55.695352    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzjtw" event={"ID":"9400c4a1-ccf5-4b45-94f8-e37079762e4f","Type":"ContainerDied","Data":"6f3433933e446ad2be68d92dd40adf255d899677bb24c1a56cb2e75de77353ae"}
Oct 29 21:45:55.697712 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:55.697697    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74bdq_259e6b95-d64c-4101-871f-4a933d4ac38f/ovn-acl-logging/0.log"
Oct 29 21:45:55.698074 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:55.698049    2579 generic.go:356] "Generic (PLEG): container finished" podID="259e6b95-d64c-4101-871f-4a933d4ac38f" containerID="e8d2e46a6f05d222f7e077887589b8e6a0b76ec5ae44079648d836889a8ddfae" exitCode=1
Oct 29 21:45:55.698127 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:55.698105    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74bdq" event={"ID":"259e6b95-d64c-4101-871f-4a933d4ac38f","Type":"ContainerStarted","Data":"e1aeb20ac7ead744eefb545c659ff1e0c1946a0f0e372a67023a86e1f08c9edd"}
Oct 29 21:45:55.698184 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:55.698171    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74bdq" event={"ID":"259e6b95-d64c-4101-871f-4a933d4ac38f","Type":"ContainerStarted","Data":"1af14dd986cb37c69ceacd9f08c7135d6e9d1f5388167a00e8f00ebdf6223826"}
Oct 29 21:45:55.698239 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:55.698226    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74bdq" event={"ID":"259e6b95-d64c-4101-871f-4a933d4ac38f","Type":"ContainerDied","Data":"e8d2e46a6f05d222f7e077887589b8e6a0b76ec5ae44079648d836889a8ddfae"}
Oct 29 21:45:55.698324 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:55.698278    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74bdq" event={"ID":"259e6b95-d64c-4101-871f-4a933d4ac38f","Type":"ContainerStarted","Data":"60430776b5f5a25a493d618d66d2dbe9a7f6ab5b565c98ae94c0a61dc315886e"}
Oct 29 21:45:56.165565 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:56.165500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/676904dc-4d4f-4bd6-b380-357d5d9a29d4-original-pull-secret\") pod \"global-pull-secret-syncer-hwq7z\" (UID: \"676904dc-4d4f-4bd6-b380-357d5d9a29d4\") " pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:45:56.165565 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:56.165552    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs\") pod \"network-metrics-daemon-l7tkk\" (UID: \"0febea83-955d-4a56-8bc4-c883a4141109\") " pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:45:56.165747 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:56.165627    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:56.165747 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:56.165684    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs podName:0febea83-955d-4a56-8bc4-c883a4141109 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:12.165660971 +0000 UTC m=+33.259289858 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs") pod "network-metrics-daemon-l7tkk" (UID: "0febea83-955d-4a56-8bc4-c883a4141109") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 21:45:56.165747 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:56.165627    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:56.165846 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:56.165754    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/676904dc-4d4f-4bd6-b380-357d5d9a29d4-original-pull-secret podName:676904dc-4d4f-4bd6-b380-357d5d9a29d4 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:12.165743201 +0000 UTC m=+33.259372090 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/676904dc-4d4f-4bd6-b380-357d5d9a29d4-original-pull-secret") pod "global-pull-secret-syncer-hwq7z" (UID: "676904dc-4d4f-4bd6-b380-357d5d9a29d4") : object "kube-system"/"original-pull-secret" not registered
Oct 29 21:45:56.266753 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:56.266731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brg9z\" (UniqueName: \"kubernetes.io/projected/f934ff4f-dadc-469b-a87e-16db95e3181f-kube-api-access-brg9z\") pod \"network-check-target-pkmzk\" (UID: \"f934ff4f-dadc-469b-a87e-16db95e3181f\") " pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:45:56.266875 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:56.266827    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 21:45:56.266875 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:56.266839    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 21:45:56.266875 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:56.266848    2579 projected.go:196] Error preparing data for projected volume kube-api-access-brg9z for pod openshift-network-diagnostics/network-check-target-pkmzk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:56.266977 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:56.266886    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f934ff4f-dadc-469b-a87e-16db95e3181f-kube-api-access-brg9z podName:f934ff4f-dadc-469b-a87e-16db95e3181f nodeName:}" failed. No retries permitted until 2026-10-29 21:46:12.266876274 +0000 UTC m=+33.360505160 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-brg9z" (UniqueName: "kubernetes.io/projected/f934ff4f-dadc-469b-a87e-16db95e3181f-kube-api-access-brg9z") pod "network-check-target-pkmzk" (UID: "f934ff4f-dadc-469b-a87e-16db95e3181f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 21:45:56.508742 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:56.508693    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:45:56.508846 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:56.508779    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkmzk" podUID="f934ff4f-dadc-469b-a87e-16db95e3181f"
Oct 29 21:45:56.703475 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:56.703449    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzjtw" event={"ID":"9400c4a1-ccf5-4b45-94f8-e37079762e4f","Type":"ContainerStarted","Data":"75e13219f0e3f572f2cdce0c516db57452721490b31c054a2ba692cf7f088dcf"}
Oct 29 21:45:56.706001 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:56.705984    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74bdq_259e6b95-d64c-4101-871f-4a933d4ac38f/ovn-acl-logging/0.log"
Oct 29 21:45:56.706360 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:56.706344    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74bdq" event={"ID":"259e6b95-d64c-4101-871f-4a933d4ac38f","Type":"ContainerStarted","Data":"f63d393acd5be4d967bf74cd56a69aae1e8ad4b82768846d16f0293c0dd9ad1a"}
Oct 29 21:45:56.706412 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:56.706395    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74bdq" event={"ID":"259e6b95-d64c-4101-871f-4a933d4ac38f","Type":"ContainerStarted","Data":"3df25a87bcc55a3ff4c3cde4c1db93697defe6aac49f83255d07cc260f946a64"}
Oct 29 21:45:56.755232 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:56.755193    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lzjtw" podStartSLOduration=8.151604361 podStartE2EDuration="17.755180201s" totalImagesPullingTime="9.60357584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:45:56.754334759 +0000 UTC m=+17.847963670" watchObservedRunningTime="2026-10-29 21:45:56.755180201 +0000 UTC m=+17.848809106"
Oct 29 21:45:57.509304 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:57.509268    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:45:57.509452 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:57.509309    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:45:57.509452 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:57.509372    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwq7z" podUID="676904dc-4d4f-4bd6-b380-357d5d9a29d4"
Oct 29 21:45:57.509571 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:57.509515    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7tkk" podUID="0febea83-955d-4a56-8bc4-c883a4141109"
Oct 29 21:45:58.509350 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:58.509320    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:45:58.509716 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:58.509408    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkmzk" podUID="f934ff4f-dadc-469b-a87e-16db95e3181f"
Oct 29 21:45:58.712633 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:58.712610    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74bdq_259e6b95-d64c-4101-871f-4a933d4ac38f/ovn-acl-logging/0.log"
Oct 29 21:45:58.712957 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:58.712934    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74bdq" event={"ID":"259e6b95-d64c-4101-871f-4a933d4ac38f","Type":"ContainerStarted","Data":"03a175e9d4d62f156d2b2999b80bdcb43128df7b674579de2587d09f71112719"}
Oct 29 21:45:59.509543 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:59.509513    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:45:59.510090 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:59.509603    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7tkk" podUID="0febea83-955d-4a56-8bc4-c883a4141109"
Oct 29 21:45:59.510090 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:45:59.509701    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:45:59.510090 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:45:59.509795    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwq7z" podUID="676904dc-4d4f-4bd6-b380-357d5d9a29d4"
Oct 29 21:46:00.509335 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:00.509138    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:46:00.509468 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:00.509414    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkmzk" podUID="f934ff4f-dadc-469b-a87e-16db95e3181f"
Oct 29 21:46:00.720051 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:00.720026    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74bdq_259e6b95-d64c-4101-871f-4a933d4ac38f/ovn-acl-logging/0.log"
Oct 29 21:46:00.720590 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:00.720379    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74bdq" event={"ID":"259e6b95-d64c-4101-871f-4a933d4ac38f","Type":"ContainerStarted","Data":"34f802b77d8684bc4adf528d4c1527fb1cbf0c119a8f7007bf3f02017a598ee3"}
Oct 29 21:46:00.720677 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:00.720660    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:00.720851 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:00.720835    2579 scope.go:122] "RemoveContainer" containerID="e8d2e46a6f05d222f7e077887589b8e6a0b76ec5ae44079648d836889a8ddfae"
Oct 29 21:46:00.736471 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:00.736451    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:01.509370 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:01.509341    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:46:01.509647 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:01.509348    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:46:01.509647 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:01.509460    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwq7z" podUID="676904dc-4d4f-4bd6-b380-357d5d9a29d4"
Oct 29 21:46:01.509647 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:01.509507    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7tkk" podUID="0febea83-955d-4a56-8bc4-c883a4141109"
Oct 29 21:46:01.725911 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:01.725888    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74bdq_259e6b95-d64c-4101-871f-4a933d4ac38f/ovn-acl-logging/0.log"
Oct 29 21:46:01.726366 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:01.726327    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-74bdq" event={"ID":"259e6b95-d64c-4101-871f-4a933d4ac38f","Type":"ContainerStarted","Data":"f85b56429b4c2776762e904be41c5fb1144201887c869a5de000db730cc0bd1b"}
Oct 29 21:46:01.726447 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:01.726411    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 21:46:01.726558 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:01.726539    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:01.742374 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:01.742349    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:01.805715 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:01.805625    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-74bdq" podStartSLOduration=8.158927237 podStartE2EDuration="22.805611746s" totalImagesPullingTime="14.646684509s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 21:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:01.805161604 +0000 UTC m=+22.898790512" watchObservedRunningTime="2026-10-29 21:46:01.805611746 +0000 UTC m=+22.899240651"
Oct 29 21:46:02.030498 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:02.030469    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hwq7z"]
Oct 29 21:46:02.030661 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:02.030628    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:46:02.030765 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:02.030737    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwq7z" podUID="676904dc-4d4f-4bd6-b380-357d5d9a29d4"
Oct 29 21:46:02.031407 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:02.031386    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pkmzk"]
Oct 29 21:46:02.031491 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:02.031478    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:46:02.031574 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:02.031555    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkmzk" podUID="f934ff4f-dadc-469b-a87e-16db95e3181f"
Oct 29 21:46:02.032929 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:02.032900    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l7tkk"]
Oct 29 21:46:02.033029 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:02.033014    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:46:02.033135 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:02.033118    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7tkk" podUID="0febea83-955d-4a56-8bc4-c883a4141109"
Oct 29 21:46:02.728429 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:02.728402    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 21:46:03.511511 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:03.511483    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:46:03.511727 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:03.511520    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:46:03.511727 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:03.511600    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7tkk" podUID="0febea83-955d-4a56-8bc4-c883a4141109"
Oct 29 21:46:03.511727 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:03.511634    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwq7z" podUID="676904dc-4d4f-4bd6-b380-357d5d9a29d4"
Oct 29 21:46:03.511727 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:03.511673    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:46:03.511880 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:03.511751    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkmzk" podUID="f934ff4f-dadc-469b-a87e-16db95e3181f"
Oct 29 21:46:03.730776 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:03.730748    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 21:46:05.509374 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:05.509328    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:46:05.509733 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:05.509328    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:46:05.509733 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:05.509447    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwq7z" podUID="676904dc-4d4f-4bd6-b380-357d5d9a29d4"
Oct 29 21:46:05.509733 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:05.509509    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkmzk" podUID="f934ff4f-dadc-469b-a87e-16db95e3181f"
Oct 29 21:46:05.509733 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:05.509337    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:46:05.509733 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:05.509617    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7tkk" podUID="0febea83-955d-4a56-8bc4-c883a4141109"
Oct 29 21:46:06.577755 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:06.577728    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:06.578112 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:06.577932    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 21:46:06.588153 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:06.588132    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:07.508998 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:07.508835    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:46:07.509112 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:07.508899    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:46:07.509112 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:07.509075    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pkmzk" podUID="f934ff4f-dadc-469b-a87e-16db95e3181f"
Oct 29 21:46:07.509200 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:07.509177    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hwq7z" podUID="676904dc-4d4f-4bd6-b380-357d5d9a29d4"
Oct 29 21:46:07.509245 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:07.508942    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:46:07.509304 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:07.509275    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l7tkk" podUID="0febea83-955d-4a56-8bc4-c883a4141109"
Oct 29 21:46:08.190913 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.190891    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeReady"
Oct 29 21:46:08.191277 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.190995    2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 29 21:46:08.243063 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.243032    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4w7w5"]
Oct 29 21:46:08.247358 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.247338    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lqjq8"]
Oct 29 21:46:08.248010 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.247979    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4w7w5"
Oct 29 21:46:08.250488 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.250468    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"]
Oct 29 21:46:08.250573 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.250548    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-lqjq8"
Oct 29 21:46:08.253128 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.253112    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k"]
Oct 29 21:46:08.253231 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.253213    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"
Oct 29 21:46:08.259057 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.259037    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-jtrf8"]
Oct 29 21:46:08.259162 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.259147    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k"
Oct 29 21:46:08.261956 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.261937    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2"]
Oct 29 21:46:08.262088 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.262072    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:08.265095 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.265076    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jdps8"]
Oct 29 21:46:08.265203 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.265185    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2"
Oct 29 21:46:08.268577 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.268558    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"]
Oct 29 21:46:08.268686 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.268672    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:08.270415 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.270394    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 29 21:46:08.272119 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.272103    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cv94k"]
Oct 29 21:46:08.272731 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.272711    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 29 21:46:08.272804 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.272771    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 29 21:46:08.273028 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.272317    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:08.275763 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.275745    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6d4df84b7f-nm9mn"]
Oct 29 21:46:08.275851 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.275835    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cv94k"
Oct 29 21:46:08.279545 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.279527    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-9np86"]
Oct 29 21:46:08.279843 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.279827    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:08.284001 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.283980    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fkzj6"]
Oct 29 21:46:08.284878 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.284854    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-9np86"
Oct 29 21:46:08.291469 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.291450    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55df9ddfdb-9gj4j"]
Oct 29 21:46:08.291603 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.291578    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fkzj6"
Oct 29 21:46:08.298253 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.298234    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-wq26q"]
Oct 29 21:46:08.298387 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.298368    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.301913 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.301893    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-547f4fc48c-tpxdx"]
Oct 29 21:46:08.302019 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.302004    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wq26q"
Oct 29 21:46:08.305126 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.305106    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-swlfh"]
Oct 29 21:46:08.305249 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.305232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:08.319707 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.319686    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dkzbq"]
Oct 29 21:46:08.319817 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.319800    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:08.322466 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.322444    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4w7w5"]
Oct 29 21:46:08.322565 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.322480    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"]
Oct 29 21:46:08.322565 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.322503    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lqjq8"]
Oct 29 21:46:08.322565 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.322525    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k"]
Oct 29 21:46:08.322565 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.322545    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-jtrf8"]
Oct 29 21:46:08.322760 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.322563    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dkzbq"
Oct 29 21:46:08.322869 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.322566    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cv94k"]
Oct 29 21:46:08.322919 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.322892    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"]
Oct 29 21:46:08.322919 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.322914    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6d4df84b7f-nm9mn"]
Oct 29 21:46:08.323054 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.322927    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fkzj6"]
Oct 29 21:46:08.323054 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.322941    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2"]
Oct 29 21:46:08.323054 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.322960    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-547f4fc48c-tpxdx"]
Oct 29 21:46:08.323054 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.322973    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jdps8"]
Oct 29 21:46:08.323054 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.322990    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-9np86"]
Oct 29 21:46:08.333329 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.333312    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55df9ddfdb-9gj4j"]
Oct 29 21:46:08.335039 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.335022    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-wq26q"]
Oct 29 21:46:08.336503 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.336481    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dkzbq"]
Oct 29 21:46:08.347611 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.347592    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-swlfh"]
Oct 29 21:46:08.361976 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.361934    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0e7f3fff-da9c-4189-bd53-5d6586ee0a4c-config\") pod \"service-ca-operator-544d749c8c-d7x4k\" (UID: \"0e7f3fff-da9c-4189-bd53-5d6586ee0a4c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k"
Oct 29 21:46:08.361976 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.361964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/30af126f-08a0-49e2-9988-50b2f8380cfe-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2\" (UID: \"30af126f-08a0-49e2-9988-50b2f8380cfe\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2"
Oct 29 21:46:08.362100 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362063    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m29ck\" (UniqueName: \"kubernetes.io/projected/0388d722-6c0b-4cf0-9f0f-7ee369004008-kube-api-access-m29ck\") pod \"cluster-samples-operator-6ddb6c8b94-nbmt9\" (UID: \"0388d722-6c0b-4cf0-9f0f-7ee369004008\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:08.362100 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362089    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6p66l\" (UID: \"1eb2f410-512d-4e3e-838a-87c1e3b47207\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"
Oct 29 21:46:08.362172 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-ca-trust-extracted\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.362172 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/72352328-6f28-47ae-9190-0176cedaed4a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9np86\" (UID: \"72352328-6f28-47ae-9190-0176cedaed4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9np86"
Oct 29 21:46:08.362242 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362203    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:08.362282 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-metrics-certs\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:08.362282 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362257    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xntk2\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-kube-api-access-xntk2\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.362360 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362352    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-snapshots\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:08.362396 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362381    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9np86\" (UID: \"72352328-6f28-47ae-9190-0176cedaed4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9np86"
Oct 29 21:46:08.362430 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362408    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thktc\" (UniqueName: \"kubernetes.io/projected/30af126f-08a0-49e2-9988-50b2f8380cfe-kube-api-access-thktc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2\" (UID: \"30af126f-08a0-49e2-9988-50b2f8380cfe\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2"
Oct 29 21:46:08.362498 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362474    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kmlt\" (UniqueName: \"kubernetes.io/projected/d172a43b-a20c-4fc3-a1cc-77a697312061-kube-api-access-8kmlt\") pod \"kas-connection-checker-685895c7c9-4w7w5\" (UID: \"d172a43b-a20c-4fc3-a1cc-77a697312061\") " pod="kube-system/kas-connection-checker-685895c7c9-4w7w5"
Oct 29 21:46:08.362566 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362532    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hs6g\" (UniqueName: \"kubernetes.io/projected/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-kube-api-access-8hs6g\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:08.362620 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362582    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-bound-sa-token\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.362673 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362633    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-image-registry-private-configuration\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.362673 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362667    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.362749 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-serving-cert\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:08.362749 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362724    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm2gz\" (UniqueName: \"kubernetes.io/projected/0e7f3fff-da9c-4189-bd53-5d6586ee0a4c-kube-api-access-dm2gz\") pod \"service-ca-operator-544d749c8c-d7x4k\" (UID: \"0e7f3fff-da9c-4189-bd53-5d6586ee0a4c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k"
Oct 29 21:46:08.362749 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-installation-pull-secrets\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.362861 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/30af126f-08a0-49e2-9988-50b2f8380cfe-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2\" (UID: \"30af126f-08a0-49e2-9988-50b2f8380cfe\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2"
Oct 29 21:46:08.362861 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nbmt9\" (UID: \"0388d722-6c0b-4cf0-9f0f-7ee369004008\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:08.362861 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362795    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whtg5\" (UniqueName: \"kubernetes.io/projected/1065c119-4756-47ad-960c-c99ee1fae963-kube-api-access-whtg5\") pod \"kas-connection-checker-685895c7c9-fkzj6\" (UID: \"1065c119-4756-47ad-960c-c99ee1fae963\") " pod="kube-system/kas-connection-checker-685895c7c9-fkzj6"
Oct 29 21:46:08.362861 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362821    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:08.362861 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362835    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-service-ca-bundle\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:08.363038 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqh8q\" (UniqueName: \"kubernetes.io/projected/a2536550-3bec-4756-b872-7b14a8eb78c3-kube-api-access-xqh8q\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:08.363038 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362952    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600-serving-cert\") pod \"console-operator-7cd56b8dbc-jdps8\" (UID: \"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:08.363038 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.362998    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-tmp\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:08.363159 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.363037    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-certificates\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.363159 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.363057    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-nbmt9\" (UID: \"0388d722-6c0b-4cf0-9f0f-7ee369004008\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:08.363159 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.363073    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2ml4\" (UniqueName: \"kubernetes.io/projected/530bea30-e132-4b1a-9c6d-e421c216060a-kube-api-access-d2ml4\") pod \"volume-data-source-validator-58c9f5f46d-cv94k\" (UID: \"530bea30-e132-4b1a-9c6d-e421c216060a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cv94k"
Oct 29 21:46:08.363159 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.363119    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600-trusted-ca\") pod \"console-operator-7cd56b8dbc-jdps8\" (UID: \"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:08.363337 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.363172    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-stats-auth\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:08.363337 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.363214    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-trusted-ca\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.363337 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.363232    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1eb2f410-512d-4e3e-838a-87c1e3b47207-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6p66l\" (UID: \"1eb2f410-512d-4e3e-838a-87c1e3b47207\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"
Oct 29 21:46:08.363337 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.363307    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-default-certificate\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:08.363511 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.363345    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0e7f3fff-da9c-4189-bd53-5d6586ee0a4c-serving-cert\") pod \"service-ca-operator-544d749c8c-d7x4k\" (UID: \"0e7f3fff-da9c-4189-bd53-5d6586ee0a4c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k"
Oct 29 21:46:08.363511 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.363368    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600-config\") pod \"console-operator-7cd56b8dbc-jdps8\" (UID: \"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:08.363511 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.363410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxt72\" (UniqueName: \"kubernetes.io/projected/73358782-c31a-4cb4-a0c2-377a1bceae4e-kube-api-access-sxt72\") pod \"kas-connection-checker-685895c7c9-lqjq8\" (UID: \"73358782-c31a-4cb4-a0c2-377a1bceae4e\") " pod="kube-system/kas-connection-checker-685895c7c9-lqjq8"
Oct 29 21:46:08.363511 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.363439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9z57\" (UniqueName: \"kubernetes.io/projected/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600-kube-api-access-v9z57\") pod \"console-operator-7cd56b8dbc-jdps8\" (UID: \"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:08.363511 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.363463    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtz7z\" (UniqueName: \"kubernetes.io/projected/1eb2f410-512d-4e3e-838a-87c1e3b47207-kube-api-access-jtz7z\") pod \"cluster-monitoring-operator-884576c57-6p66l\" (UID: \"1eb2f410-512d-4e3e-838a-87c1e3b47207\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"
Oct 29 21:46:08.371191 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.371170    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:46:08.371280 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.371194    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 29 21:46:08.371428 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.371414    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 29 21:46:08.371488 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.371416    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 29 21:46:08.371488 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.371416    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 29 21:46:08.371488 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.371420    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 29 21:46:08.371578 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.371529    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 29 21:46:08.371740 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.371724    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 29 21:46:08.371782 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.371774    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 29 21:46:08.371959 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.371945    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 29 21:46:08.371959 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.371954    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 29 21:46:08.372078 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.371989    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 29 21:46:08.372078 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.372021    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 29 21:46:08.372184 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.372137    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 29 21:46:08.372305 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.372265    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:46:08.372367 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.372307    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:46:08.372367 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.372326    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 29 21:46:08.372367 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.372332    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 29 21:46:08.372367 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.372354    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 29 21:46:08.372614 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.372603    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:46:08.372662 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.372646    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 29 21:46:08.372708 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.372677    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 29 21:46:08.384616 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.384598    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 21:46:08.384616 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.384599    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 21:46:08.384998 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.384976    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 29 21:46:08.385074 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.385017    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 29 21:46:08.385338 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.385322    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:46:08.385414 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.385369    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 29 21:46:08.385469 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.385415    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 29 21:46:08.385502 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.385469    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 29 21:46:08.386032 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.386017    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 29 21:46:08.386083 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.386062    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 29 21:46:08.386083 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.386067    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 29 21:46:08.386142 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.386122    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 29 21:46:08.386317 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.386305    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 29 21:46:08.386371 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.386340    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 29 21:46:08.386371 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.386348    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 29 21:46:08.386586 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.386572    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 29 21:46:08.386634 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.386591    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 29 21:46:08.386736 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.386717    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 29 21:46:08.386736 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.386738    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 29 21:46:08.393440 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.393419    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-hx8mh\""
Oct 29 21:46:08.393440 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.393424    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-t7vd6\""
Oct 29 21:46:08.393557 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.393479    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rh2k6\""
Oct 29 21:46:08.394718 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.394705    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vdqjz\""
Oct 29 21:46:08.394755 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.394738    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-ldpp7\""
Oct 29 21:46:08.394755 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.394748    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-p5fxq\""
Oct 29 21:46:08.394820 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.394738    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wm9k9\""
Oct 29 21:46:08.394860 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.394845    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cgft7\""
Oct 29 21:46:08.394860 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.394852    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-pl8bq\""
Oct 29 21:46:08.394946 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.394884    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-zphp7\""
Oct 29 21:46:08.394946 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.394905    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-hjsdq\""
Oct 29 21:46:08.394946 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.394911    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-sxnlk\""
Oct 29 21:46:08.394946 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.394918    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jtcl2\""
Oct 29 21:46:08.398088 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.398058    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 29 21:46:08.398973 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.398961    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 29 21:46:08.400932 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.400920    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 29 21:46:08.401736 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.401717    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 29 21:46:08.401782 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.401754    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 29 21:46:08.464437 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.464415    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/72352328-6f28-47ae-9190-0176cedaed4a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9np86\" (UID: \"72352328-6f28-47ae-9190-0176cedaed4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9np86"
Oct 29 21:46:08.464531 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.464443    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:08.464531 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.464462    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-metrics-certs\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:08.464531 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.464486    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xntk2\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-kube-api-access-xntk2\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.464672 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.464584    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle podName:a2536550-3bec-4756-b872-7b14a8eb78c3 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:08.964568891 +0000 UTC m=+30.058197796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle") pod "router-default-6d4df84b7f-nm9mn" (UID: "a2536550-3bec-4756-b872-7b14a8eb78c3") : configmap references non-existent config key: service-ca.crt
Oct 29 21:46:08.464672 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.464597    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 29 21:46:08.464672 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.464618    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-snapshots\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:08.464672 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.464654    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-metrics-certs podName:a2536550-3bec-4756-b872-7b14a8eb78c3 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:08.964635664 +0000 UTC m=+30.058264554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-metrics-certs") pod "router-default-6d4df84b7f-nm9mn" (UID: "a2536550-3bec-4756-b872-7b14a8eb78c3") : secret "router-metrics-certs-default" not found
Oct 29 21:46:08.464888 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.464677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9np86\" (UID: \"72352328-6f28-47ae-9190-0176cedaed4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9np86"
Oct 29 21:46:08.464888 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.464709    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thktc\" (UniqueName: \"kubernetes.io/projected/30af126f-08a0-49e2-9988-50b2f8380cfe-kube-api-access-thktc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2\" (UID: \"30af126f-08a0-49e2-9988-50b2f8380cfe\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2"
Oct 29 21:46:08.464888 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.464738    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kmlt\" (UniqueName: \"kubernetes.io/projected/d172a43b-a20c-4fc3-a1cc-77a697312061-kube-api-access-8kmlt\") pod \"kas-connection-checker-685895c7c9-4w7w5\" (UID: \"d172a43b-a20c-4fc3-a1cc-77a697312061\") " pod="kube-system/kas-connection-checker-685895c7c9-4w7w5"
Oct 29 21:46:08.464888 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.464763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hs6g\" (UniqueName: \"kubernetes.io/projected/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-kube-api-access-8hs6g\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:08.464888 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.464784    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 29 21:46:08.464888 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.464832    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert podName:72352328-6f28-47ae-9190-0176cedaed4a nodeName:}" failed. No retries permitted until 2026-10-29 21:46:08.964812408 +0000 UTC m=+30.058441308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9np86" (UID: "72352328-6f28-47ae-9190-0176cedaed4a") : secret "networking-console-plugin-cert" not found
Oct 29 21:46:08.464888 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.464787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-bound-sa-token\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.465211 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.464888    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-image-registry-private-configuration\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.465211 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.464931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.465211 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.464957    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-serving-cert\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:08.465211 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.464985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm2gz\" (UniqueName: \"kubernetes.io/projected/0e7f3fff-da9c-4189-bd53-5d6586ee0a4c-kube-api-access-dm2gz\") pod \"service-ca-operator-544d749c8c-d7x4k\" (UID: \"0e7f3fff-da9c-4189-bd53-5d6586ee0a4c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k"
Oct 29 21:46:08.465211 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.465012    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-installation-pull-secrets\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.465211 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.465016    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 21:46:08.465211 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.465030    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55df9ddfdb-9gj4j: secret "image-registry-tls" not found
Oct 29 21:46:08.465211 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.465037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/30af126f-08a0-49e2-9988-50b2f8380cfe-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2\" (UID: \"30af126f-08a0-49e2-9988-50b2f8380cfe\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2"
Oct 29 21:46:08.465211 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.465065    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls podName:81d7056d-8cf4-4058-a8fa-1731a0b93ff6 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:08.965052366 +0000 UTC m=+30.058681268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls") pod "image-registry-55df9ddfdb-9gj4j" (UID: "81d7056d-8cf4-4058-a8fa-1731a0b93ff6") : secret "image-registry-tls" not found
Oct 29 21:46:08.465211 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.465109    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4b5fdb27-5000-4cf8-a025-13ae5389d17f-ca-trust-extracted\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:08.465211 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.465135    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69xwz\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-kube-api-access-69xwz\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:08.465211 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.465177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nbmt9\" (UID: \"0388d722-6c0b-4cf0-9f0f-7ee369004008\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:08.465211 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.465198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whtg5\" (UniqueName: \"kubernetes.io/projected/1065c119-4756-47ad-960c-c99ee1fae963-kube-api-access-whtg5\") pod \"kas-connection-checker-685895c7c9-fkzj6\" (UID: \"1065c119-4756-47ad-960c-c99ee1fae963\") " pod="kube-system/kas-connection-checker-685895c7c9-fkzj6"
Oct 29 21:46:08.465709 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.465234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:08.465709 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.465311    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/72352328-6f28-47ae-9190-0176cedaed4a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9np86\" (UID: \"72352328-6f28-47ae-9190-0176cedaed4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9np86"
Oct 29 21:46:08.465709 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.465317    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-snapshots\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:08.465709 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.465342    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 29 21:46:08.465709 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.465352    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-service-ca-bundle\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:08.465709 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.465392    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls podName:0388d722-6c0b-4cf0-9f0f-7ee369004008 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:08.965379379 +0000 UTC m=+30.059008296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-nbmt9" (UID: "0388d722-6c0b-4cf0-9f0f-7ee369004008") : secret "samples-operator-tls" not found
Oct 29 21:46:08.465709 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.465439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07a537c1-669a-4a02-bc1f-f09ea24e47ca-cert\") pod \"ingress-canary-dkzbq\" (UID: \"07a537c1-669a-4a02-bc1f-f09ea24e47ca\") " pod="openshift-ingress-canary/ingress-canary-dkzbq"
Oct 29 21:46:08.465709 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.465468    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4b5fdb27-5000-4cf8-a025-13ae5389d17f-installation-pull-secrets\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:08.466056 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.465788    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqh8q\" (UniqueName: \"kubernetes.io/projected/a2536550-3bec-4756-b872-7b14a8eb78c3-kube-api-access-xqh8q\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:08.466439 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.466416    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-service-ca-bundle\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:08.466677 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.466562    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-bound-sa-token\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:08.466677 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.466647    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:08.466802 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.466780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4b5fdb27-5000-4cf8-a025-13ae5389d17f-image-registry-private-configuration\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:08.466879 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.466861    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600-serving-cert\") pod \"console-operator-7cd56b8dbc-jdps8\" (UID: \"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:08.467025 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.467006    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-tmp\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:08.467128 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.467115    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-certificates\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:08.467221 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.467206    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/99c9881d-05b3-4953-b484-3ce51520b84a-config-volume\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:08.467358 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.467335    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99c9881d-05b3-4953-b484-3ce51520b84a-metrics-tls\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:08.467433 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.467389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-certificates\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.467487 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.467431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-nbmt9\" (UID: \"0388d722-6c0b-4cf0-9f0f-7ee369004008\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:08.467487 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.467466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2ml4\" (UniqueName: \"kubernetes.io/projected/530bea30-e132-4b1a-9c6d-e421c216060a-kube-api-access-d2ml4\") pod \"volume-data-source-validator-58c9f5f46d-cv94k\" (UID: \"530bea30-e132-4b1a-9c6d-e421c216060a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cv94k"
Oct 29 21:46:08.469900 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.467817    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-tmp\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:08.469900 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.467869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600-trusted-ca\") pod \"console-operator-7cd56b8dbc-jdps8\" (UID: \"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:08.469900 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.467902    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxjln\" (UniqueName: \"kubernetes.io/projected/07a537c1-669a-4a02-bc1f-f09ea24e47ca-kube-api-access-qxjln\") pod \"ingress-canary-dkzbq\" (UID: \"07a537c1-669a-4a02-bc1f-f09ea24e47ca\") " pod="openshift-ingress-canary/ingress-canary-dkzbq"
Oct 29 21:46:08.469900 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.467938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-stats-auth\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:08.469900 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.467973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-trusted-ca\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.469900 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.468006    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1eb2f410-512d-4e3e-838a-87c1e3b47207-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6p66l\" (UID: \"1eb2f410-512d-4e3e-838a-87c1e3b47207\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"
Oct 29 21:46:08.469900 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.468063    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:08.469900 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.468081    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-certificates\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.469900 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.468102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-default-certificate\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:08.469900 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.468134    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z46kb\" (UniqueName: \"kubernetes.io/projected/99c9881d-05b3-4953-b484-3ce51520b84a-kube-api-access-z46kb\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:08.469900 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.468168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0e7f3fff-da9c-4189-bd53-5d6586ee0a4c-serving-cert\") pod \"service-ca-operator-544d749c8c-d7x4k\" (UID: \"0e7f3fff-da9c-4189-bd53-5d6586ee0a4c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k"
Oct 29 21:46:08.469900 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.468198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600-config\") pod \"console-operator-7cd56b8dbc-jdps8\" (UID: \"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:08.469900 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.468238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxt72\" (UniqueName: \"kubernetes.io/projected/73358782-c31a-4cb4-a0c2-377a1bceae4e-kube-api-access-sxt72\") pod \"kas-connection-checker-685895c7c9-lqjq8\" (UID: \"73358782-c31a-4cb4-a0c2-377a1bceae4e\") " pod="kube-system/kas-connection-checker-685895c7c9-lqjq8"
Oct 29 21:46:08.469900 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.468266    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9z57\" (UniqueName: \"kubernetes.io/projected/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600-kube-api-access-v9z57\") pod \"console-operator-7cd56b8dbc-jdps8\" (UID: \"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:08.469900 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.468317    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtz7z\" (UniqueName: \"kubernetes.io/projected/1eb2f410-512d-4e3e-838a-87c1e3b47207-kube-api-access-jtz7z\") pod \"cluster-monitoring-operator-884576c57-6p66l\" (UID: \"1eb2f410-512d-4e3e-838a-87c1e3b47207\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"
Oct 29 21:46:08.469900 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.468342    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv58t\" (UniqueName: \"kubernetes.io/projected/0816e7fb-46c8-49d0-9c07-36fc49188b84-kube-api-access-lv58t\") pod \"network-check-source-6f78d59f99-wq26q\" (UID: \"0816e7fb-46c8-49d0-9c07-36fc49188b84\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wq26q"
Oct 29 21:46:08.470627 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.468375    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b5fdb27-5000-4cf8-a025-13ae5389d17f-trusted-ca\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:08.470627 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.468433    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0e7f3fff-da9c-4189-bd53-5d6586ee0a4c-config\") pod \"service-ca-operator-544d749c8c-d7x4k\" (UID: \"0e7f3fff-da9c-4189-bd53-5d6586ee0a4c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k"
Oct 29 21:46:08.470627 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.468462    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/30af126f-08a0-49e2-9988-50b2f8380cfe-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2\" (UID: \"30af126f-08a0-49e2-9988-50b2f8380cfe\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2"
Oct 29 21:46:08.470627 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.468493    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m29ck\" (UniqueName: \"kubernetes.io/projected/0388d722-6c0b-4cf0-9f0f-7ee369004008-kube-api-access-m29ck\") pod \"cluster-samples-operator-6ddb6c8b94-nbmt9\" (UID: \"0388d722-6c0b-4cf0-9f0f-7ee369004008\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:08.470627 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.468519    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6p66l\" (UID: \"1eb2f410-512d-4e3e-838a-87c1e3b47207\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"
Oct 29 21:46:08.470627 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.468546    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c9881d-05b3-4953-b484-3ce51520b84a-tmp-dir\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:08.470627 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.468563    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-nbmt9\" (UID: \"0388d722-6c0b-4cf0-9f0f-7ee369004008\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:08.470627 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.468577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-ca-trust-extracted\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.470627 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.469099    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-trusted-ca\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.470627 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.469474    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1eb2f410-512d-4e3e-838a-87c1e3b47207-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6p66l\" (UID: \"1eb2f410-512d-4e3e-838a-87c1e3b47207\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"
Oct 29 21:46:08.470627 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.469557    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0e7f3fff-da9c-4189-bd53-5d6586ee0a4c-config\") pod \"service-ca-operator-544d749c8c-d7x4k\" (UID: \"0e7f3fff-da9c-4189-bd53-5d6586ee0a4c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k"
Oct 29 21:46:08.470627 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.470234    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 29 21:46:08.470627 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.470311    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls podName:1eb2f410-512d-4e3e-838a-87c1e3b47207 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:08.970269598 +0000 UTC m=+30.063898501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6p66l" (UID: "1eb2f410-512d-4e3e-838a-87c1e3b47207") : secret "cluster-monitoring-operator-tls" not found
Oct 29 21:46:08.470627 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.470360    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-ca-trust-extracted\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.470627 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.470362    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600-config\") pod \"console-operator-7cd56b8dbc-jdps8\" (UID: \"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:08.471243 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.470489    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/30af126f-08a0-49e2-9988-50b2f8380cfe-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2\" (UID: \"30af126f-08a0-49e2-9988-50b2f8380cfe\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2"
Oct 29 21:46:08.471243 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.471152    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600-trusted-ca\") pod \"console-operator-7cd56b8dbc-jdps8\" (UID: \"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:08.471359 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.471341    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-image-registry-private-configuration\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.473647 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.473626    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-stats-auth\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:08.473741 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.473628    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600-serving-cert\") pod \"console-operator-7cd56b8dbc-jdps8\" (UID: \"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:08.473741 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.473660    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-installation-pull-secrets\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.474653 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.474610    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/30af126f-08a0-49e2-9988-50b2f8380cfe-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2\" (UID: \"30af126f-08a0-49e2-9988-50b2f8380cfe\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2"
Oct 29 21:46:08.475257 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.475227    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-serving-cert\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:08.475445 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.475420    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0e7f3fff-da9c-4189-bd53-5d6586ee0a4c-serving-cert\") pod \"service-ca-operator-544d749c8c-d7x4k\" (UID: \"0e7f3fff-da9c-4189-bd53-5d6586ee0a4c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k"
Oct 29 21:46:08.475758 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.475739    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-default-certificate\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:08.494889 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.494860    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thktc\" (UniqueName: \"kubernetes.io/projected/30af126f-08a0-49e2-9988-50b2f8380cfe-kube-api-access-thktc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2\" (UID: \"30af126f-08a0-49e2-9988-50b2f8380cfe\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2"
Oct 29 21:46:08.495726 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.495700    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hs6g\" (UniqueName: \"kubernetes.io/projected/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-kube-api-access-8hs6g\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:08.496401 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.496381    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xntk2\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-kube-api-access-xntk2\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.496401 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.496396    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-bound-sa-token\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.496643 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.496625    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm2gz\" (UniqueName: \"kubernetes.io/projected/0e7f3fff-da9c-4189-bd53-5d6586ee0a4c-kube-api-access-dm2gz\") pod \"service-ca-operator-544d749c8c-d7x4k\" (UID: \"0e7f3fff-da9c-4189-bd53-5d6586ee0a4c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k"
Oct 29 21:46:08.509257 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.508724    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxt72\" (UniqueName: \"kubernetes.io/projected/73358782-c31a-4cb4-a0c2-377a1bceae4e-kube-api-access-sxt72\") pod \"kas-connection-checker-685895c7c9-lqjq8\" (UID: \"73358782-c31a-4cb4-a0c2-377a1bceae4e\") " pod="kube-system/kas-connection-checker-685895c7c9-lqjq8"
Oct 29 21:46:08.509257 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.509181    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kmlt\" (UniqueName: \"kubernetes.io/projected/d172a43b-a20c-4fc3-a1cc-77a697312061-kube-api-access-8kmlt\") pod \"kas-connection-checker-685895c7c9-4w7w5\" (UID: \"d172a43b-a20c-4fc3-a1cc-77a697312061\") " pod="kube-system/kas-connection-checker-685895c7c9-4w7w5"
Oct 29 21:46:08.509257 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.509207    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whtg5\" (UniqueName: \"kubernetes.io/projected/1065c119-4756-47ad-960c-c99ee1fae963-kube-api-access-whtg5\") pod \"kas-connection-checker-685895c7c9-fkzj6\" (UID: \"1065c119-4756-47ad-960c-c99ee1fae963\") " pod="kube-system/kas-connection-checker-685895c7c9-fkzj6"
Oct 29 21:46:08.509452 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.509327    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m29ck\" (UniqueName: \"kubernetes.io/projected/0388d722-6c0b-4cf0-9f0f-7ee369004008-kube-api-access-m29ck\") pod \"cluster-samples-operator-6ddb6c8b94-nbmt9\" (UID: \"0388d722-6c0b-4cf0-9f0f-7ee369004008\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:08.509452 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.509371    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2ml4\" (UniqueName: \"kubernetes.io/projected/530bea30-e132-4b1a-9c6d-e421c216060a-kube-api-access-d2ml4\") pod \"volume-data-source-validator-58c9f5f46d-cv94k\" (UID: \"530bea30-e132-4b1a-9c6d-e421c216060a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cv94k"
Oct 29 21:46:08.509601 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.509579    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtz7z\" (UniqueName: \"kubernetes.io/projected/1eb2f410-512d-4e3e-838a-87c1e3b47207-kube-api-access-jtz7z\") pod \"cluster-monitoring-operator-884576c57-6p66l\" (UID: \"1eb2f410-512d-4e3e-838a-87c1e3b47207\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"
Oct 29 21:46:08.510832 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.510811    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqh8q\" (UniqueName: \"kubernetes.io/projected/a2536550-3bec-4756-b872-7b14a8eb78c3-kube-api-access-xqh8q\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:08.512186 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.512169    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9z57\" (UniqueName: \"kubernetes.io/projected/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600-kube-api-access-v9z57\") pod \"console-operator-7cd56b8dbc-jdps8\" (UID: \"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:08.564235 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.564215    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4w7w5"
Oct 29 21:46:08.569199 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.569183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69xwz\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-kube-api-access-69xwz\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:08.569299 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.569225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07a537c1-669a-4a02-bc1f-f09ea24e47ca-cert\") pod \"ingress-canary-dkzbq\" (UID: \"07a537c1-669a-4a02-bc1f-f09ea24e47ca\") " pod="openshift-ingress-canary/ingress-canary-dkzbq"
Oct 29 21:46:08.569299 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.569241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4b5fdb27-5000-4cf8-a025-13ae5389d17f-installation-pull-secrets\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:08.569299 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.569258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-bound-sa-token\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:08.569443 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.569318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4b5fdb27-5000-4cf8-a025-13ae5389d17f-image-registry-private-configuration\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:08.569443 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.569348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-certificates\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:08.569443 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.569371    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/99c9881d-05b3-4953-b484-3ce51520b84a-config-volume\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:08.569443 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.569407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99c9881d-05b3-4953-b484-3ce51520b84a-metrics-tls\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:08.569642 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.569469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxjln\" (UniqueName: \"kubernetes.io/projected/07a537c1-669a-4a02-bc1f-f09ea24e47ca-kube-api-access-qxjln\") pod \"ingress-canary-dkzbq\" (UID: \"07a537c1-669a-4a02-bc1f-f09ea24e47ca\") " pod="openshift-ingress-canary/ingress-canary-dkzbq"
Oct 29 21:46:08.569642 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.569520    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:08.569642 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.569557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z46kb\" (UniqueName: \"kubernetes.io/projected/99c9881d-05b3-4953-b484-3ce51520b84a-kube-api-access-z46kb\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:08.569642 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.569596    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv58t\" (UniqueName: \"kubernetes.io/projected/0816e7fb-46c8-49d0-9c07-36fc49188b84-kube-api-access-lv58t\") pod \"network-check-source-6f78d59f99-wq26q\" (UID: \"0816e7fb-46c8-49d0-9c07-36fc49188b84\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wq26q"
Oct 29 21:46:08.569642 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.569636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b5fdb27-5000-4cf8-a025-13ae5389d17f-trusted-ca\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:08.569870 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.569680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c9881d-05b3-4953-b484-3ce51520b84a-tmp-dir\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:08.569870 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.569787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4b5fdb27-5000-4cf8-a025-13ae5389d17f-ca-trust-extracted\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:08.569870 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.569801    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 21:46:08.569870 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.569857    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c9881d-05b3-4953-b484-3ce51520b84a-metrics-tls podName:99c9881d-05b3-4953-b484-3ce51520b84a nodeName:}" failed. No retries permitted until 2026-10-29 21:46:09.069839505 +0000 UTC m=+30.163468396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99c9881d-05b3-4953-b484-3ce51520b84a-metrics-tls") pod "dns-default-swlfh" (UID: "99c9881d-05b3-4953-b484-3ce51520b84a") : secret "dns-default-metrics-tls" not found
Oct 29 21:46:08.570070 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.569933    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 21:46:08.570070 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.569946    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-547f4fc48c-tpxdx: secret "image-registry-tls" not found
Oct 29 21:46:08.570070 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.569960    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/99c9881d-05b3-4953-b484-3ce51520b84a-config-volume\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:08.570070 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.569980    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls podName:4b5fdb27-5000-4cf8-a025-13ae5389d17f nodeName:}" failed. No retries permitted until 2026-10-29 21:46:09.069968707 +0000 UTC m=+30.163597595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls") pod "image-registry-547f4fc48c-tpxdx" (UID: "4b5fdb27-5000-4cf8-a025-13ae5389d17f") : secret "image-registry-tls" not found
Oct 29 21:46:08.570070 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.569981    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-certificates\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:08.570296 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.570147    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4b5fdb27-5000-4cf8-a025-13ae5389d17f-ca-trust-extracted\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:08.570296 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.570242    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c9881d-05b3-4953-b484-3ce51520b84a-tmp-dir\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:08.571026 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.571000    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b5fdb27-5000-4cf8-a025-13ae5389d17f-trusted-ca\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:08.571557 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.571537    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07a537c1-669a-4a02-bc1f-f09ea24e47ca-cert\") pod \"ingress-canary-dkzbq\" (UID: \"07a537c1-669a-4a02-bc1f-f09ea24e47ca\") " pod="openshift-ingress-canary/ingress-canary-dkzbq"
Oct 29 21:46:08.572019 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.571998    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-lqjq8"
Oct 29 21:46:08.572219 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.572200    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4b5fdb27-5000-4cf8-a025-13ae5389d17f-image-registry-private-configuration\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:08.572282 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.572225    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4b5fdb27-5000-4cf8-a025-13ae5389d17f-installation-pull-secrets\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:08.589738 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.589705    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k"
Oct 29 21:46:08.595991 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.595879    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z46kb\" (UniqueName: \"kubernetes.io/projected/99c9881d-05b3-4953-b484-3ce51520b84a-kube-api-access-z46kb\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:08.595991 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.595934    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-bound-sa-token\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:08.596485 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.596461    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxjln\" (UniqueName: \"kubernetes.io/projected/07a537c1-669a-4a02-bc1f-f09ea24e47ca-kube-api-access-qxjln\") pod \"ingress-canary-dkzbq\" (UID: \"07a537c1-669a-4a02-bc1f-f09ea24e47ca\") " pod="openshift-ingress-canary/ingress-canary-dkzbq"
Oct 29 21:46:08.596615 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.596517    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69xwz\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-kube-api-access-69xwz\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:08.596615 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.596550    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv58t\" (UniqueName: \"kubernetes.io/projected/0816e7fb-46c8-49d0-9c07-36fc49188b84-kube-api-access-lv58t\") pod \"network-check-source-6f78d59f99-wq26q\" (UID: \"0816e7fb-46c8-49d0-9c07-36fc49188b84\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wq26q"
Oct 29 21:46:08.597535 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.597515    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:08.606450 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.606430    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2"
Oct 29 21:46:08.620253 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.620098    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:08.631099 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.630914    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cv94k"
Oct 29 21:46:08.659202 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.658869    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fkzj6"
Oct 29 21:46:08.668650 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.668588    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wq26q"
Oct 29 21:46:08.691893 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.691459    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dkzbq"
Oct 29 21:46:08.904684 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.904583    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-lqjq8"]
Oct 29 21:46:08.905146 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.905124    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4w7w5"]
Oct 29 21:46:08.907752 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:46:08.907727    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd172a43b_a20c_4fc3_a1cc_77a697312061.slice/crio-f682692a9619f3ebdb75e9843c1c09182742e713349b7cfb4b6d083bf870f951 WatchSource:0}: Error finding container f682692a9619f3ebdb75e9843c1c09182742e713349b7cfb4b6d083bf870f951: Status 404 returned error can't find the container with id f682692a9619f3ebdb75e9843c1c09182742e713349b7cfb4b6d083bf870f951
Oct 29 21:46:08.908269 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:46:08.908246    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73358782_c31a_4cb4_a0c2_377a1bceae4e.slice/crio-8b24e062c3af897f8f9d7ea821a287050e2df1c7c14c1e7818c2ca7814d8edac WatchSource:0}: Error finding container 8b24e062c3af897f8f9d7ea821a287050e2df1c7c14c1e7818c2ca7814d8edac: Status 404 returned error can't find the container with id 8b24e062c3af897f8f9d7ea821a287050e2df1c7c14c1e7818c2ca7814d8edac
Oct 29 21:46:08.947568 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.947408    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k"]
Oct 29 21:46:08.954395 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:46:08.954360    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e7f3fff_da9c_4189_bd53_5d6586ee0a4c.slice/crio-056b9baf976e19e2257192d2af06ff59d4fab0e77ec5e24badf9543b14134834 WatchSource:0}: Error finding container 056b9baf976e19e2257192d2af06ff59d4fab0e77ec5e24badf9543b14134834: Status 404 returned error can't find the container with id 056b9baf976e19e2257192d2af06ff59d4fab0e77ec5e24badf9543b14134834
Oct 29 21:46:08.958426 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.957061    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-jtrf8"]
Oct 29 21:46:08.961759 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:46:08.961730    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55bbb5ff_1be8_4b7b_a12f_4dd1a3db7e97.slice/crio-46a570e581b4fb9a1bc8a245abf0cc7cd235810158021bc29081243a3aa83256 WatchSource:0}: Error finding container 46a570e581b4fb9a1bc8a245abf0cc7cd235810158021bc29081243a3aa83256: Status 404 returned error can't find the container with id 46a570e581b4fb9a1bc8a245abf0cc7cd235810158021bc29081243a3aa83256
Oct 29 21:46:08.967865 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.967839    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2"]
Oct 29 21:46:08.971525 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:46:08.971477    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30af126f_08a0_49e2_9988_50b2f8380cfe.slice/crio-26f431d8cc18618290f28183fa37a140bb0f72bcd41b7cb644247ff93ae84d30 WatchSource:0}: Error finding container 26f431d8cc18618290f28183fa37a140bb0f72bcd41b7cb644247ff93ae84d30: Status 404 returned error can't find the container with id 26f431d8cc18618290f28183fa37a140bb0f72bcd41b7cb644247ff93ae84d30
Oct 29 21:46:08.973264 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.972870    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6p66l\" (UID: \"1eb2f410-512d-4e3e-838a-87c1e3b47207\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"
Oct 29 21:46:08.973264 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.972920    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:08.973264 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.972951    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-metrics-certs\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:08.973264 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.972986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9np86\" (UID: \"72352328-6f28-47ae-9190-0176cedaed4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9np86"
Oct 29 21:46:08.973264 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.973023    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:08.973264 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:08.973055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nbmt9\" (UID: \"0388d722-6c0b-4cf0-9f0f-7ee369004008\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:08.974384 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.974052    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 29 21:46:08.974384 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.974076    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 29 21:46:08.974384 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.974117    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls podName:1eb2f410-512d-4e3e-838a-87c1e3b47207 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:09.974096529 +0000 UTC m=+31.067725439 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6p66l" (UID: "1eb2f410-512d-4e3e-838a-87c1e3b47207") : secret "cluster-monitoring-operator-tls" not found
Oct 29 21:46:08.974384 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.974138    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert podName:72352328-6f28-47ae-9190-0176cedaed4a nodeName:}" failed. No retries permitted until 2026-10-29 21:46:09.974127243 +0000 UTC m=+31.067756136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9np86" (UID: "72352328-6f28-47ae-9190-0176cedaed4a") : secret "networking-console-plugin-cert" not found
Oct 29 21:46:08.974384 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.974208    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 21:46:08.974384 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.974219    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55df9ddfdb-9gj4j: secret "image-registry-tls" not found
Oct 29 21:46:08.974384 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.974251    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls podName:81d7056d-8cf4-4058-a8fa-1731a0b93ff6 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:09.974241015 +0000 UTC m=+31.067869917 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls") pod "image-registry-55df9ddfdb-9gj4j" (UID: "81d7056d-8cf4-4058-a8fa-1731a0b93ff6") : secret "image-registry-tls" not found
Oct 29 21:46:08.974384 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.974352    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle podName:a2536550-3bec-4756-b872-7b14a8eb78c3 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:09.974341143 +0000 UTC m=+31.067970049 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle") pod "router-default-6d4df84b7f-nm9mn" (UID: "a2536550-3bec-4756-b872-7b14a8eb78c3") : configmap references non-existent config key: service-ca.crt
Oct 29 21:46:08.974798 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.974467    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 29 21:46:08.974798 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.974525    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls podName:0388d722-6c0b-4cf0-9f0f-7ee369004008 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:09.974505425 +0000 UTC m=+31.068134324 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-nbmt9" (UID: "0388d722-6c0b-4cf0-9f0f-7ee369004008") : secret "samples-operator-tls" not found
Oct 29 21:46:08.974798 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.974583    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 29 21:46:08.974798 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:08.974626    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-metrics-certs podName:a2536550-3bec-4756-b872-7b14a8eb78c3 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:09.974614034 +0000 UTC m=+31.068242935 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-metrics-certs") pod "router-default-6d4df84b7f-nm9mn" (UID: "a2536550-3bec-4756-b872-7b14a8eb78c3") : secret "router-metrics-certs-default" not found
Oct 29 21:46:09.020662 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.020643    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-jdps8"]
Oct 29 21:46:09.023504 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:46:09.023485    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ed52a4_5ca3_4c5c_a2b5_f4bf6fb28600.slice/crio-259bea6e5a944368b40bbd724270b44065932a6167fa1e8ccac33ec171b40c55 WatchSource:0}: Error finding container 259bea6e5a944368b40bbd724270b44065932a6167fa1e8ccac33ec171b40c55: Status 404 returned error can't find the container with id 259bea6e5a944368b40bbd724270b44065932a6167fa1e8ccac33ec171b40c55
Oct 29 21:46:09.046635 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.046611    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cv94k"]
Oct 29 21:46:09.049694 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:46:09.049674    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod530bea30_e132_4b1a_9c6d_e421c216060a.slice/crio-31582fbc125b37e50cc345dbd42e7047341b76afd096dd265541a5a55e9b5b94 WatchSource:0}: Error finding container 31582fbc125b37e50cc345dbd42e7047341b76afd096dd265541a5a55e9b5b94: Status 404 returned error can't find the container with id 31582fbc125b37e50cc345dbd42e7047341b76afd096dd265541a5a55e9b5b94
Oct 29 21:46:09.074528 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.074509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99c9881d-05b3-4953-b484-3ce51520b84a-metrics-tls\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:09.074638 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.074560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:09.074700 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:09.074686    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 21:46:09.074751 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:09.074705    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 21:46:09.074751 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:09.074715    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-547f4fc48c-tpxdx: secret "image-registry-tls" not found
Oct 29 21:46:09.074822 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:09.074750    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c9881d-05b3-4953-b484-3ce51520b84a-metrics-tls podName:99c9881d-05b3-4953-b484-3ce51520b84a nodeName:}" failed. No retries permitted until 2026-10-29 21:46:10.074730276 +0000 UTC m=+31.168359166 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99c9881d-05b3-4953-b484-3ce51520b84a-metrics-tls") pod "dns-default-swlfh" (UID: "99c9881d-05b3-4953-b484-3ce51520b84a") : secret "dns-default-metrics-tls" not found
Oct 29 21:46:09.074822 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:09.074770    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls podName:4b5fdb27-5000-4cf8-a025-13ae5389d17f nodeName:}" failed. No retries permitted until 2026-10-29 21:46:10.074760224 +0000 UTC m=+31.168389115 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls") pod "image-registry-547f4fc48c-tpxdx" (UID: "4b5fdb27-5000-4cf8-a025-13ae5389d17f") : secret "image-registry-tls" not found
Oct 29 21:46:09.090713 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.090696    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dkzbq"]
Oct 29 21:46:09.090869 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:46:09.090839    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1065c119_4756_47ad_960c_c99ee1fae963.slice/crio-188b2721cbc4bb7917d2e8302a064622c484d612c3c48f8cedfc66fe3201f1ec WatchSource:0}: Error finding container 188b2721cbc4bb7917d2e8302a064622c484d612c3c48f8cedfc66fe3201f1ec: Status 404 returned error can't find the container with id 188b2721cbc4bb7917d2e8302a064622c484d612c3c48f8cedfc66fe3201f1ec
Oct 29 21:46:09.091106 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.091088    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fkzj6"]
Oct 29 21:46:09.091799 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:46:09.091764    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a537c1_669a_4a02_bc1f_f09ea24e47ca.slice/crio-4dbbd0f8ac94c46de1949f712095f105b2dc8de0840e7561ee87316e9aec8483 WatchSource:0}: Error finding container 4dbbd0f8ac94c46de1949f712095f105b2dc8de0840e7561ee87316e9aec8483: Status 404 returned error can't find the container with id 4dbbd0f8ac94c46de1949f712095f105b2dc8de0840e7561ee87316e9aec8483
Oct 29 21:46:09.100033 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.100014    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-wq26q"]
Oct 29 21:46:09.105082 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:46:09.105059    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0816e7fb_46c8_49d0_9c07_36fc49188b84.slice/crio-f00d7456ca3983460851aa61fdc8cfafa316a32c96c906e9f24ad72c5eedc0e9 WatchSource:0}: Error finding container f00d7456ca3983460851aa61fdc8cfafa316a32c96c906e9f24ad72c5eedc0e9: Status 404 returned error can't find the container with id f00d7456ca3983460851aa61fdc8cfafa316a32c96c906e9f24ad72c5eedc0e9
Oct 29 21:46:09.511184 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.511111    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:46:09.511823 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.511581    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:46:09.511823 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.511791    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:46:09.529350 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.529004    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 21:46:09.529350 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.529092    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 21:46:09.529350 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.529151    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hzxgj\""
Oct 29 21:46:09.529350 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.529224    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4sldf\""
Oct 29 21:46:09.529350 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.529324    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jsffp\""
Oct 29 21:46:09.750762 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.750660    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wq26q" event={"ID":"0816e7fb-46c8-49d0-9c07-36fc49188b84","Type":"ContainerStarted","Data":"f00d7456ca3983460851aa61fdc8cfafa316a32c96c906e9f24ad72c5eedc0e9"}
Oct 29 21:46:09.754153 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.754114    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4w7w5" event={"ID":"d172a43b-a20c-4fc3-a1cc-77a697312061","Type":"ContainerStarted","Data":"4842671ad52f71ae0dbbb70a3e311177b59231bfb4eb782ec0b0fb440f435467"}
Oct 29 21:46:09.754275 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.754164    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4w7w5" event={"ID":"d172a43b-a20c-4fc3-a1cc-77a697312061","Type":"ContainerStarted","Data":"f682692a9619f3ebdb75e9843c1c09182742e713349b7cfb4b6d083bf870f951"}
Oct 29 21:46:09.757007 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.756977    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8" event={"ID":"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600","Type":"ContainerStarted","Data":"259bea6e5a944368b40bbd724270b44065932a6167fa1e8ccac33ec171b40c55"}
Oct 29 21:46:09.760449 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.760396    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k" event={"ID":"0e7f3fff-da9c-4189-bd53-5d6586ee0a4c","Type":"ContainerStarted","Data":"056b9baf976e19e2257192d2af06ff59d4fab0e77ec5e24badf9543b14134834"}
Oct 29 21:46:09.764615 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.764552    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dkzbq" event={"ID":"07a537c1-669a-4a02-bc1f-f09ea24e47ca","Type":"ContainerStarted","Data":"4dbbd0f8ac94c46de1949f712095f105b2dc8de0840e7561ee87316e9aec8483"}
Oct 29 21:46:09.767696 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.767643    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2" event={"ID":"30af126f-08a0-49e2-9988-50b2f8380cfe","Type":"ContainerStarted","Data":"26f431d8cc18618290f28183fa37a140bb0f72bcd41b7cb644247ff93ae84d30"}
Oct 29 21:46:09.769278 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.769255    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-jtrf8" event={"ID":"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97","Type":"ContainerStarted","Data":"46a570e581b4fb9a1bc8a245abf0cc7cd235810158021bc29081243a3aa83256"}
Oct 29 21:46:09.775715 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.775162    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-lqjq8" event={"ID":"73358782-c31a-4cb4-a0c2-377a1bceae4e","Type":"ContainerStarted","Data":"aa24fa7e98e80416ca1398b6fdbe5f98b92fd6fc209bfccf37c344e0bd612420"}
Oct 29 21:46:09.775715 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.775203    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-lqjq8" event={"ID":"73358782-c31a-4cb4-a0c2-377a1bceae4e","Type":"ContainerStarted","Data":"8b24e062c3af897f8f9d7ea821a287050e2df1c7c14c1e7818c2ca7814d8edac"}
Oct 29 21:46:09.779917 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.779544    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fkzj6" event={"ID":"1065c119-4756-47ad-960c-c99ee1fae963","Type":"ContainerStarted","Data":"3bd0435fcce8cd389dc41a73f4412dd4b81062c2bac5ed3177e26fa97f067783"}
Oct 29 21:46:09.779917 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.779582    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fkzj6" event={"ID":"1065c119-4756-47ad-960c-c99ee1fae963","Type":"ContainerStarted","Data":"188b2721cbc4bb7917d2e8302a064622c484d612c3c48f8cedfc66fe3201f1ec"}
Oct 29 21:46:09.782877 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.782853    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cv94k" event={"ID":"530bea30-e132-4b1a-9c6d-e421c216060a","Type":"ContainerStarted","Data":"31582fbc125b37e50cc345dbd42e7047341b76afd096dd265541a5a55e9b5b94"}
Oct 29 21:46:09.800357 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.799551    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-4w7w5" podStartSLOduration=294.79953661 podStartE2EDuration="4m54.79953661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:09.798634792 +0000 UTC m=+30.892263702" watchObservedRunningTime="2026-10-29 21:46:09.79953661 +0000 UTC m=+30.893165519"
Oct 29 21:46:09.847189 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.847143    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-fkzj6" podStartSLOduration=294.847126354 podStartE2EDuration="4m54.847126354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:09.845614532 +0000 UTC m=+30.939243442" watchObservedRunningTime="2026-10-29 21:46:09.847126354 +0000 UTC m=+30.940755263"
Oct 29 21:46:09.982722 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.981954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6p66l\" (UID: \"1eb2f410-512d-4e3e-838a-87c1e3b47207\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"
Oct 29 21:46:09.982722 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.982011    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:09.982722 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.982038    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-metrics-certs\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:09.982722 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.982076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9np86\" (UID: \"72352328-6f28-47ae-9190-0176cedaed4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9np86"
Oct 29 21:46:09.982722 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.982114    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:09.982722 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:09.982128    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 29 21:46:09.982722 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:09.982196    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls podName:1eb2f410-512d-4e3e-838a-87c1e3b47207 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:11.982175829 +0000 UTC m=+33.075804734 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6p66l" (UID: "1eb2f410-512d-4e3e-838a-87c1e3b47207") : secret "cluster-monitoring-operator-tls" not found
Oct 29 21:46:09.982722 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:09.982211    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 29 21:46:09.982722 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:09.982245    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls podName:0388d722-6c0b-4cf0-9f0f-7ee369004008 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:11.982234431 +0000 UTC m=+33.075863332 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-nbmt9" (UID: "0388d722-6c0b-4cf0-9f0f-7ee369004008") : secret "samples-operator-tls" not found
Oct 29 21:46:09.982722 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:09.982365    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle podName:a2536550-3bec-4756-b872-7b14a8eb78c3 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:11.982355362 +0000 UTC m=+33.075984262 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle") pod "router-default-6d4df84b7f-nm9mn" (UID: "a2536550-3bec-4756-b872-7b14a8eb78c3") : configmap references non-existent config key: service-ca.crt
Oct 29 21:46:09.982722 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:09.982424    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 29 21:46:09.982722 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:09.982456    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-metrics-certs podName:a2536550-3bec-4756-b872-7b14a8eb78c3 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:11.982444513 +0000 UTC m=+33.076073416 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-metrics-certs") pod "router-default-6d4df84b7f-nm9mn" (UID: "a2536550-3bec-4756-b872-7b14a8eb78c3") : secret "router-metrics-certs-default" not found
Oct 29 21:46:09.982722 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:09.982509    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 29 21:46:09.982722 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:09.982540    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert podName:72352328-6f28-47ae-9190-0176cedaed4a nodeName:}" failed. No retries permitted until 2026-10-29 21:46:11.982529991 +0000 UTC m=+33.076158892 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9np86" (UID: "72352328-6f28-47ae-9190-0176cedaed4a") : secret "networking-console-plugin-cert" not found
Oct 29 21:46:09.983548 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:09.982541    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nbmt9\" (UID: \"0388d722-6c0b-4cf0-9f0f-7ee369004008\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:09.983548 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:09.982603    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 21:46:09.983548 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:09.982613    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55df9ddfdb-9gj4j: secret "image-registry-tls" not found
Oct 29 21:46:09.983548 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:09.982640    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls podName:81d7056d-8cf4-4058-a8fa-1731a0b93ff6 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:11.982631316 +0000 UTC m=+33.076260217 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls") pod "image-registry-55df9ddfdb-9gj4j" (UID: "81d7056d-8cf4-4058-a8fa-1731a0b93ff6") : secret "image-registry-tls" not found
Oct 29 21:46:10.083988 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:10.083903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:10.084134 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:10.084116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99c9881d-05b3-4953-b484-3ce51520b84a-metrics-tls\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:10.084493 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:10.084281    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 21:46:10.084493 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:10.084312    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 21:46:10.084493 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:10.084333    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-547f4fc48c-tpxdx: secret "image-registry-tls" not found
Oct 29 21:46:10.084493 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:10.084363    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c9881d-05b3-4953-b484-3ce51520b84a-metrics-tls podName:99c9881d-05b3-4953-b484-3ce51520b84a nodeName:}" failed. No retries permitted until 2026-10-29 21:46:12.084346788 +0000 UTC m=+33.177975677 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99c9881d-05b3-4953-b484-3ce51520b84a-metrics-tls") pod "dns-default-swlfh" (UID: "99c9881d-05b3-4953-b484-3ce51520b84a") : secret "dns-default-metrics-tls" not found
Oct 29 21:46:10.084493 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:10.084382    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls podName:4b5fdb27-5000-4cf8-a025-13ae5389d17f nodeName:}" failed. No retries permitted until 2026-10-29 21:46:12.084372844 +0000 UTC m=+33.178001731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls") pod "image-registry-547f4fc48c-tpxdx" (UID: "4b5fdb27-5000-4cf8-a025-13ae5389d17f") : secret "image-registry-tls" not found
Oct 29 21:46:11.795375 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:11.795326    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cv94k" event={"ID":"530bea30-e132-4b1a-9c6d-e421c216060a","Type":"ContainerStarted","Data":"032e4532ae5caa0e8f901a462263edf75fc8d1d00caa8250d6ca1ddd5a779f32"}
Oct 29 21:46:11.840033 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:11.839967    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cv94k" podStartSLOduration=268.464635777 podStartE2EDuration="4m30.839949575s" totalImagesPullingTime="2.375313798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:11.839457723 +0000 UTC m=+32.933086644" watchObservedRunningTime="2026-10-29 21:46:11.839949575 +0000 UTC m=+32.933578484"
Oct 29 21:46:11.841519 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:11.841238    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-lqjq8" podStartSLOduration=296.841221899 podStartE2EDuration="4m56.841221899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:09.89121365 +0000 UTC m=+30.984842559" watchObservedRunningTime="2026-10-29 21:46:11.841221899 +0000 UTC m=+32.934850814"
Oct 29 21:46:12.004511 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:12.004473    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6p66l\" (UID: \"1eb2f410-512d-4e3e-838a-87c1e3b47207\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"
Oct 29 21:46:12.004676 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:12.004533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:12.004676 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:12.004564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-metrics-certs\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:12.004676 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:12.004600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9np86\" (UID: \"72352328-6f28-47ae-9190-0176cedaed4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9np86"
Oct 29 21:46:12.004676 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:12.004637    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:12.004676 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:12.004672    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nbmt9\" (UID: \"0388d722-6c0b-4cf0-9f0f-7ee369004008\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:12.004862 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:12.004842    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 29 21:46:12.004923 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:12.004904    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls podName:0388d722-6c0b-4cf0-9f0f-7ee369004008 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:16.004885255 +0000 UTC m=+37.098514161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-nbmt9" (UID: "0388d722-6c0b-4cf0-9f0f-7ee369004008") : secret "samples-operator-tls" not found
Oct 29 21:46:12.005670 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:12.005306    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 29 21:46:12.005670 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:12.005355    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls podName:1eb2f410-512d-4e3e-838a-87c1e3b47207 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:16.005341238 +0000 UTC m=+37.098970141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6p66l" (UID: "1eb2f410-512d-4e3e-838a-87c1e3b47207") : secret "cluster-monitoring-operator-tls" not found
Oct 29 21:46:12.005670 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:12.005422    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle podName:a2536550-3bec-4756-b872-7b14a8eb78c3 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:16.005411523 +0000 UTC m=+37.099040423 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle") pod "router-default-6d4df84b7f-nm9mn" (UID: "a2536550-3bec-4756-b872-7b14a8eb78c3") : configmap references non-existent config key: service-ca.crt
Oct 29 21:46:12.005670 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:12.005493    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 29 21:46:12.005670 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:12.005526    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-metrics-certs podName:a2536550-3bec-4756-b872-7b14a8eb78c3 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:16.005516283 +0000 UTC m=+37.099145188 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-metrics-certs") pod "router-default-6d4df84b7f-nm9mn" (UID: "a2536550-3bec-4756-b872-7b14a8eb78c3") : secret "router-metrics-certs-default" not found
Oct 29 21:46:12.005670 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:12.005578    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 29 21:46:12.005670 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:12.005608    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert podName:72352328-6f28-47ae-9190-0176cedaed4a nodeName:}" failed. No retries permitted until 2026-10-29 21:46:16.005598435 +0000 UTC m=+37.099227337 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9np86" (UID: "72352328-6f28-47ae-9190-0176cedaed4a") : secret "networking-console-plugin-cert" not found
Oct 29 21:46:12.006210 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:12.006127    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 21:46:12.006210 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:12.006147    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55df9ddfdb-9gj4j: secret "image-registry-tls" not found
Oct 29 21:46:12.006210 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:12.006193    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls podName:81d7056d-8cf4-4058-a8fa-1731a0b93ff6 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:16.006179036 +0000 UTC m=+37.099807926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls") pod "image-registry-55df9ddfdb-9gj4j" (UID: "81d7056d-8cf4-4058-a8fa-1731a0b93ff6") : secret "image-registry-tls" not found
Oct 29 21:46:12.105645 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:12.105607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:12.105828 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:12.105810    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99c9881d-05b3-4953-b484-3ce51520b84a-metrics-tls\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:12.105969 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:12.105954    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 21:46:12.106040 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:12.106014    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c9881d-05b3-4953-b484-3ce51520b84a-metrics-tls podName:99c9881d-05b3-4953-b484-3ce51520b84a nodeName:}" failed. No retries permitted until 2026-10-29 21:46:16.105996203 +0000 UTC m=+37.199625103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99c9881d-05b3-4953-b484-3ce51520b84a-metrics-tls") pod "dns-default-swlfh" (UID: "99c9881d-05b3-4953-b484-3ce51520b84a") : secret "dns-default-metrics-tls" not found
Oct 29 21:46:12.106101 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:12.106086    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 21:46:12.106101 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:12.106097    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-547f4fc48c-tpxdx: secret "image-registry-tls" not found
Oct 29 21:46:12.106195 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:12.106131    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls podName:4b5fdb27-5000-4cf8-a025-13ae5389d17f nodeName:}" failed. No retries permitted until 2026-10-29 21:46:16.106119879 +0000 UTC m=+37.199748782 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls") pod "image-registry-547f4fc48c-tpxdx" (UID: "4b5fdb27-5000-4cf8-a025-13ae5389d17f") : secret "image-registry-tls" not found
Oct 29 21:46:12.207203 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:12.207167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs\") pod \"network-metrics-daemon-l7tkk\" (UID: \"0febea83-955d-4a56-8bc4-c883a4141109\") " pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:46:12.207469 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:12.207215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/676904dc-4d4f-4bd6-b380-357d5d9a29d4-original-pull-secret\") pod \"global-pull-secret-syncer-hwq7z\" (UID: \"676904dc-4d4f-4bd6-b380-357d5d9a29d4\") " pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:46:12.207469 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:12.207423    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 29 21:46:12.207469 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:12.207471    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs podName:0febea83-955d-4a56-8bc4-c883a4141109 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:44.207458595 +0000 UTC m=+65.301087485 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs") pod "network-metrics-daemon-l7tkk" (UID: "0febea83-955d-4a56-8bc4-c883a4141109") : secret "metrics-daemon-secret" not found
Oct 29 21:46:12.220682 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:12.220649    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/676904dc-4d4f-4bd6-b380-357d5d9a29d4-original-pull-secret\") pod \"global-pull-secret-syncer-hwq7z\" (UID: \"676904dc-4d4f-4bd6-b380-357d5d9a29d4\") " pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:46:12.255226 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:12.255190    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:46:12.308774 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:12.308710    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brg9z\" (UniqueName: \"kubernetes.io/projected/f934ff4f-dadc-469b-a87e-16db95e3181f-kube-api-access-brg9z\") pod \"network-check-target-pkmzk\" (UID: \"f934ff4f-dadc-469b-a87e-16db95e3181f\") " pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:46:12.312560 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:12.312537    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brg9z\" (UniqueName: \"kubernetes.io/projected/f934ff4f-dadc-469b-a87e-16db95e3181f-kube-api-access-brg9z\") pod \"network-check-target-pkmzk\" (UID: \"f934ff4f-dadc-469b-a87e-16db95e3181f\") " pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:46:12.576475 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:12.576145    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:46:12.616123 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:12.616096    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hwq7z"]
Oct 29 21:46:12.631360 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:46:12.631334    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod676904dc_4d4f_4bd6_b380_357d5d9a29d4.slice/crio-4e14abf65dd39ec088575e0d8b9e2d5ca91f63b57b2f9080b3f7af18d1b1cdf0 WatchSource:0}: Error finding container 4e14abf65dd39ec088575e0d8b9e2d5ca91f63b57b2f9080b3f7af18d1b1cdf0: Status 404 returned error can't find the container with id 4e14abf65dd39ec088575e0d8b9e2d5ca91f63b57b2f9080b3f7af18d1b1cdf0
Oct 29 21:46:12.802274 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:12.802202    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hwq7z" event={"ID":"676904dc-4d4f-4bd6-b380-357d5d9a29d4","Type":"ContainerStarted","Data":"4e14abf65dd39ec088575e0d8b9e2d5ca91f63b57b2f9080b3f7af18d1b1cdf0"}
Oct 29 21:46:12.931491 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:12.931455    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pkmzk"]
Oct 29 21:46:13.008825 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:46:13.008785    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf934ff4f_dadc_469b_a87e_16db95e3181f.slice/crio-1b6a087600cbdcb9ce02b7a4e146e0dd49a26d361b2bfd58000df97cd080550b WatchSource:0}: Error finding container 1b6a087600cbdcb9ce02b7a4e146e0dd49a26d361b2bfd58000df97cd080550b: Status 404 returned error can't find the container with id 1b6a087600cbdcb9ce02b7a4e146e0dd49a26d361b2bfd58000df97cd080550b
Oct 29 21:46:13.810064 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:13.809976    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2" event={"ID":"30af126f-08a0-49e2-9988-50b2f8380cfe","Type":"ContainerStarted","Data":"36c03b38b676b32d5a3119ed532ca94892dd79907fecbdb13928cb375f8b8f49"}
Oct 29 21:46:13.813858 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:13.813831    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/0.log"
Oct 29 21:46:13.813996 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:13.813878    2579 generic.go:356] "Generic (PLEG): container finished" podID="f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600" containerID="dbcde5446a62c4ed4177e55c477b309ad03eef287bdce9557961bb24651e9c90" exitCode=255
Oct 29 21:46:13.814182 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:13.814123    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8" event={"ID":"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600","Type":"ContainerDied","Data":"dbcde5446a62c4ed4177e55c477b309ad03eef287bdce9557961bb24651e9c90"}
Oct 29 21:46:13.814256 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:13.814194    2579 scope.go:122] "RemoveContainer" containerID="dbcde5446a62c4ed4177e55c477b309ad03eef287bdce9557961bb24651e9c90"
Oct 29 21:46:13.816599 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:13.816547    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k" event={"ID":"0e7f3fff-da9c-4189-bd53-5d6586ee0a4c","Type":"ContainerStarted","Data":"c246dd289b948d0e2ab7032057d4957492710b93e75b7cac748d124d08a45c2a"}
Oct 29 21:46:13.818657 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:13.818610    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pkmzk" event={"ID":"f934ff4f-dadc-469b-a87e-16db95e3181f","Type":"ContainerStarted","Data":"1b6a087600cbdcb9ce02b7a4e146e0dd49a26d361b2bfd58000df97cd080550b"}
Oct 29 21:46:13.858949 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:13.858889    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2" podStartSLOduration=268.254410655 podStartE2EDuration="4m32.858873927s" totalImagesPullingTime="4.604463272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:13.858226132 +0000 UTC m=+34.951855044" watchObservedRunningTime="2026-10-29 21:46:13.858873927 +0000 UTC m=+34.952502837"
Oct 29 21:46:13.907010 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:13.906959    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k" podStartSLOduration=268.296378263 podStartE2EDuration="4m32.906941819s" totalImagesPullingTime="4.610563556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:13.904977781 +0000 UTC m=+34.998606694" watchObservedRunningTime="2026-10-29 21:46:13.906941819 +0000 UTC m=+35.000570729"
Oct 29 21:46:14.609312 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:14.608434    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gs6zr"]
Oct 29 21:46:14.625780 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:14.625135    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gs6zr"
Oct 29 21:46:14.635011 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:14.634959    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gs6zr"]
Oct 29 21:46:14.653592 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:14.653572    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 29 21:46:14.653719 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:14.653710    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 29 21:46:14.670529 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:14.670414    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hxxfk\""
Oct 29 21:46:14.735822 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:14.735780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52fgx\" (UniqueName: \"kubernetes.io/projected/5c0cf3cb-ed1a-453c-a3a4-9289b557ae3d-kube-api-access-52fgx\") pod \"migrator-66df5d496b-gs6zr\" (UID: \"5c0cf3cb-ed1a-453c-a3a4-9289b557ae3d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gs6zr"
Oct 29 21:46:14.824985 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:14.824528    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dkzbq" event={"ID":"07a537c1-669a-4a02-bc1f-f09ea24e47ca","Type":"ContainerStarted","Data":"c957a075faa55eed4fe6c65a3808e97aa5d5f8d11ff03cc3cd98194179cd94ce"}
Oct 29 21:46:14.827253 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:14.827221    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-jtrf8" event={"ID":"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97","Type":"ContainerStarted","Data":"883f2e198a3926fa52689dc2aeaedbeb14c07e38b701b3ca6d1440f81cab02a3"}
Oct 29 21:46:14.829080 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:14.829062    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/1.log"
Oct 29 21:46:14.829483 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:14.829466    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/0.log"
Oct 29 21:46:14.829579 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:14.829497    2579 generic.go:356] "Generic (PLEG): container finished" podID="f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600" containerID="b727bfa5392f31848f4d9bdd2173e4c65b562208ae7f10f517776dd67c5132f8" exitCode=255
Oct 29 21:46:14.830050 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:14.830035    2579 scope.go:122] "RemoveContainer" containerID="b727bfa5392f31848f4d9bdd2173e4c65b562208ae7f10f517776dd67c5132f8"
Oct 29 21:46:14.830212 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:14.830192    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jdps8_openshift-console-operator(f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8" podUID="f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600"
Oct 29 21:46:14.830451 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:14.830428    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8" event={"ID":"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600","Type":"ContainerDied","Data":"b727bfa5392f31848f4d9bdd2173e4c65b562208ae7f10f517776dd67c5132f8"}
Oct 29 21:46:14.830577 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:14.830477    2579 scope.go:122] "RemoveContainer" containerID="dbcde5446a62c4ed4177e55c477b309ad03eef287bdce9557961bb24651e9c90"
Oct 29 21:46:14.836547 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:14.836515    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52fgx\" (UniqueName: \"kubernetes.io/projected/5c0cf3cb-ed1a-453c-a3a4-9289b557ae3d-kube-api-access-52fgx\") pod \"migrator-66df5d496b-gs6zr\" (UID: \"5c0cf3cb-ed1a-453c-a3a4-9289b557ae3d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gs6zr"
Oct 29 21:46:14.864684 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:14.864659    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52fgx\" (UniqueName: \"kubernetes.io/projected/5c0cf3cb-ed1a-453c-a3a4-9289b557ae3d-kube-api-access-52fgx\") pod \"migrator-66df5d496b-gs6zr\" (UID: \"5c0cf3cb-ed1a-453c-a3a4-9289b557ae3d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gs6zr"
Oct 29 21:46:14.870688 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:14.870639    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dkzbq" podStartSLOduration=1.496151206 podStartE2EDuration="6.87062421s" totalImagesPullingTime="5.374473004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:14.869798376 +0000 UTC m=+35.963427285" watchObservedRunningTime="2026-10-29 21:46:14.87062421 +0000 UTC m=+35.964253118"
Oct 29 21:46:14.918728 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:14.918672    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-jtrf8" podStartSLOduration=268.862869255 podStartE2EDuration="4m33.918655451s" totalImagesPullingTime="5.055786196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:41:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:14.918273858 +0000 UTC m=+36.011902764" watchObservedRunningTime="2026-10-29 21:46:14.918655451 +0000 UTC m=+36.012284363"
Oct 29 21:46:14.961358 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:14.956485    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gs6zr"
Oct 29 21:46:15.297909 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.297780    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-gs6zr"]
Oct 29 21:46:15.301599 ip-10-0-129-146 kubenswrapper[2579]: W1029 21:46:15.301571    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c0cf3cb_ed1a_453c_a3a4_9289b557ae3d.slice/crio-e8169e439f336e2ac9872ddad9e96cf2394c0d583a96b876ebdbc2b2efa91e32 WatchSource:0}: Error finding container e8169e439f336e2ac9872ddad9e96cf2394c0d583a96b876ebdbc2b2efa91e32: Status 404 returned error can't find the container with id e8169e439f336e2ac9872ddad9e96cf2394c0d583a96b876ebdbc2b2efa91e32
Oct 29 21:46:15.564839 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.563516    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dtgwl"]
Oct 29 21:46:15.593969 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.593940    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dtgwl"]
Oct 29 21:46:15.594113 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.594044    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:15.691749 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.691716    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 29 21:46:15.691749 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.691737    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 29 21:46:15.691965 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.691829    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w9rdw\""
Oct 29 21:46:15.748213 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.748182    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d45cab6-698a-4a61-be2c-58fe7d41c55c-data-volume\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:15.748420 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.748344    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:15.748593 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.748571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mppj9\" (UniqueName: \"kubernetes.io/projected/8d45cab6-698a-4a61-be2c-58fe7d41c55c-kube-api-access-mppj9\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:15.748646 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.748630    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d45cab6-698a-4a61-be2c-58fe7d41c55c-crio-socket\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:15.748698 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.748683    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d45cab6-698a-4a61-be2c-58fe7d41c55c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:15.837278 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.836042    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/1.log"
Oct 29 21:46:15.837278 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.836524    2579 scope.go:122] "RemoveContainer" containerID="b727bfa5392f31848f4d9bdd2173e4c65b562208ae7f10f517776dd67c5132f8"
Oct 29 21:46:15.837278 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:15.836738    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jdps8_openshift-console-operator(f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8" podUID="f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600"
Oct 29 21:46:15.839318 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.839238    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gs6zr" event={"ID":"5c0cf3cb-ed1a-453c-a3a4-9289b557ae3d","Type":"ContainerStarted","Data":"e8169e439f336e2ac9872ddad9e96cf2394c0d583a96b876ebdbc2b2efa91e32"}
Oct 29 21:46:15.844688 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.844643    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wq26q" event={"ID":"0816e7fb-46c8-49d0-9c07-36fc49188b84","Type":"ContainerStarted","Data":"9663a5dca5ce65129609a13a374335b72c668e568c433add0e71a0207e2c8d50"}
Oct 29 21:46:15.850150 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.850121    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mppj9\" (UniqueName: \"kubernetes.io/projected/8d45cab6-698a-4a61-be2c-58fe7d41c55c-kube-api-access-mppj9\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:15.850257 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.850174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d45cab6-698a-4a61-be2c-58fe7d41c55c-crio-socket\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:15.850257 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.850202    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d45cab6-698a-4a61-be2c-58fe7d41c55c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:15.850384 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.850256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d45cab6-698a-4a61-be2c-58fe7d41c55c-data-volume\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:15.850384 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.850344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:15.850501 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:15.850486    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:15.850547 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:15.850542    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls podName:8d45cab6-698a-4a61-be2c-58fe7d41c55c nodeName:}" failed. No retries permitted until 2026-10-29 21:46:16.350527401 +0000 UTC m=+37.444156304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtgwl" (UID: "8d45cab6-698a-4a61-be2c-58fe7d41c55c") : secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:15.850984 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.850951    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d45cab6-698a-4a61-be2c-58fe7d41c55c-crio-socket\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:15.851094 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:15.851034    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 29 21:46:15.851094 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:15.851079    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d45cab6-698a-4a61-be2c-58fe7d41c55c-kube-rbac-proxy-cm podName:8d45cab6-698a-4a61-be2c-58fe7d41c55c nodeName:}" failed. No retries permitted until 2026-10-29 21:46:16.351064813 +0000 UTC m=+37.444693712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8d45cab6-698a-4a61-be2c-58fe7d41c55c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtgwl" (UID: "8d45cab6-698a-4a61-be2c-58fe7d41c55c") : configmap "kube-rbac-proxy" not found
Oct 29 21:46:15.851385 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.851367    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d45cab6-698a-4a61-be2c-58fe7d41c55c-data-volume\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:15.877446 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.877425    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mppj9\" (UniqueName: \"kubernetes.io/projected/8d45cab6-698a-4a61-be2c-58fe7d41c55c-kube-api-access-mppj9\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:15.936938 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:15.936890    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wq26q" podStartSLOduration=265.104924059 podStartE2EDuration="4m30.936877481s" totalImagesPullingTime="5.831953422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:15.936769122 +0000 UTC m=+37.030398030" watchObservedRunningTime="2026-10-29 21:46:15.936877481 +0000 UTC m=+37.030506380"
Oct 29 21:46:16.052763 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:16.052729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9np86\" (UID: \"72352328-6f28-47ae-9190-0176cedaed4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9np86"
Oct 29 21:46:16.052916 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:16.052798    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:16.052916 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:16.052842    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nbmt9\" (UID: \"0388d722-6c0b-4cf0-9f0f-7ee369004008\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:16.053021 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:16.052974    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6p66l\" (UID: \"1eb2f410-512d-4e3e-838a-87c1e3b47207\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"
Oct 29 21:46:16.053021 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:16.053009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:16.053090 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:16.053029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-metrics-certs\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:16.053175 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:16.053160    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 29 21:46:16.053232 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:16.053221    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-metrics-certs podName:a2536550-3bec-4756-b872-7b14a8eb78c3 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:24.05320862 +0000 UTC m=+45.146837507 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-metrics-certs") pod "router-default-6d4df84b7f-nm9mn" (UID: "a2536550-3bec-4756-b872-7b14a8eb78c3") : secret "router-metrics-certs-default" not found
Oct 29 21:46:16.053528 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:16.053516    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 29 21:46:16.053568 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:16.053552    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert podName:72352328-6f28-47ae-9190-0176cedaed4a nodeName:}" failed. No retries permitted until 2026-10-29 21:46:24.053541993 +0000 UTC m=+45.147170880 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9np86" (UID: "72352328-6f28-47ae-9190-0176cedaed4a") : secret "networking-console-plugin-cert" not found
Oct 29 21:46:16.053622 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:16.053606    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 21:46:16.053622 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:16.053616    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55df9ddfdb-9gj4j: secret "image-registry-tls" not found
Oct 29 21:46:16.053717 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:16.053643    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls podName:81d7056d-8cf4-4058-a8fa-1731a0b93ff6 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:24.05363572 +0000 UTC m=+45.147264618 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls") pod "image-registry-55df9ddfdb-9gj4j" (UID: "81d7056d-8cf4-4058-a8fa-1731a0b93ff6") : secret "image-registry-tls" not found
Oct 29 21:46:16.053717 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:16.053689    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 29 21:46:16.053797 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:16.053720    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls podName:0388d722-6c0b-4cf0-9f0f-7ee369004008 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:24.053710536 +0000 UTC m=+45.147339437 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-nbmt9" (UID: "0388d722-6c0b-4cf0-9f0f-7ee369004008") : secret "samples-operator-tls" not found
Oct 29 21:46:16.053797 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:16.053757    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 29 21:46:16.053797 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:16.053778    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls podName:1eb2f410-512d-4e3e-838a-87c1e3b47207 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:24.053771946 +0000 UTC m=+45.147400831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6p66l" (UID: "1eb2f410-512d-4e3e-838a-87c1e3b47207") : secret "cluster-monitoring-operator-tls" not found
Oct 29 21:46:16.053942 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:16.053818    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle podName:a2536550-3bec-4756-b872-7b14a8eb78c3 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:24.053812973 +0000 UTC m=+45.147441864 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle") pod "router-default-6d4df84b7f-nm9mn" (UID: "a2536550-3bec-4756-b872-7b14a8eb78c3") : configmap references non-existent config key: service-ca.crt
Oct 29 21:46:16.154573 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:16.154543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99c9881d-05b3-4953-b484-3ce51520b84a-metrics-tls\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:16.154715 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:16.154611    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:16.154715 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:16.154687    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 21:46:16.154818 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:16.154769    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c9881d-05b3-4953-b484-3ce51520b84a-metrics-tls podName:99c9881d-05b3-4953-b484-3ce51520b84a nodeName:}" failed. No retries permitted until 2026-10-29 21:46:24.154750231 +0000 UTC m=+45.248379132 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99c9881d-05b3-4953-b484-3ce51520b84a-metrics-tls") pod "dns-default-swlfh" (UID: "99c9881d-05b3-4953-b484-3ce51520b84a") : secret "dns-default-metrics-tls" not found
Oct 29 21:46:16.154818 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:16.154768    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 21:46:16.154818 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:16.154791    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-547f4fc48c-tpxdx: secret "image-registry-tls" not found
Oct 29 21:46:16.154917 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:16.154836    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls podName:4b5fdb27-5000-4cf8-a025-13ae5389d17f nodeName:}" failed. No retries permitted until 2026-10-29 21:46:24.154821962 +0000 UTC m=+45.248450864 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls") pod "image-registry-547f4fc48c-tpxdx" (UID: "4b5fdb27-5000-4cf8-a025-13ae5389d17f") : secret "image-registry-tls" not found
Oct 29 21:46:16.357055 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:16.356934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:16.357055 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:16.357027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d45cab6-698a-4a61-be2c-58fe7d41c55c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:16.357197 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:16.357109    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 29 21:46:16.357197 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:16.357146    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:16.357197 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:16.357175    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d45cab6-698a-4a61-be2c-58fe7d41c55c-kube-rbac-proxy-cm podName:8d45cab6-698a-4a61-be2c-58fe7d41c55c nodeName:}" failed. No retries permitted until 2026-10-29 21:46:17.357159534 +0000 UTC m=+38.450788434 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8d45cab6-698a-4a61-be2c-58fe7d41c55c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtgwl" (UID: "8d45cab6-698a-4a61-be2c-58fe7d41c55c") : configmap "kube-rbac-proxy" not found
Oct 29 21:46:16.357197 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:16.357198    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls podName:8d45cab6-698a-4a61-be2c-58fe7d41c55c nodeName:}" failed. No retries permitted until 2026-10-29 21:46:17.357185338 +0000 UTC m=+38.450814236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtgwl" (UID: "8d45cab6-698a-4a61-be2c-58fe7d41c55c") : secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:16.849984 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:16.849847    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pkmzk" event={"ID":"f934ff4f-dadc-469b-a87e-16db95e3181f","Type":"ContainerStarted","Data":"e99377c5a671f306e5f6932adc9c952219513e95d802a193570ca9195e11c0a8"}
Oct 29 21:46:16.896556 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:16.896506    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pkmzk" podStartSLOduration=33.549509612 podStartE2EDuration="36.896489957s" totalImagesPullingTime="3.346980345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:16.89596216 +0000 UTC m=+37.989591071" watchObservedRunningTime="2026-10-29 21:46:16.896489957 +0000 UTC m=+37.990118866"
Oct 29 21:46:17.365116 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:17.365078    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d45cab6-698a-4a61-be2c-58fe7d41c55c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:17.365338 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:17.365200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:17.365338 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:17.365206    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 29 21:46:17.365338 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:17.365271    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d45cab6-698a-4a61-be2c-58fe7d41c55c-kube-rbac-proxy-cm podName:8d45cab6-698a-4a61-be2c-58fe7d41c55c nodeName:}" failed. No retries permitted until 2026-10-29 21:46:19.365255703 +0000 UTC m=+40.458884602 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8d45cab6-698a-4a61-be2c-58fe7d41c55c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtgwl" (UID: "8d45cab6-698a-4a61-be2c-58fe7d41c55c") : configmap "kube-rbac-proxy" not found
Oct 29 21:46:17.365487 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:17.365355    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:17.365487 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:17.365408    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls podName:8d45cab6-698a-4a61-be2c-58fe7d41c55c nodeName:}" failed. No retries permitted until 2026-10-29 21:46:19.36539351 +0000 UTC m=+40.459022416 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtgwl" (UID: "8d45cab6-698a-4a61-be2c-58fe7d41c55c") : secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:17.855019 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:17.854976    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gs6zr" event={"ID":"5c0cf3cb-ed1a-453c-a3a4-9289b557ae3d","Type":"ContainerStarted","Data":"133045e40a2fcc6c94116fd495b1e91d6429488d0549e404d361031c58321c49"}
Oct 29 21:46:17.855448 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:17.855207    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:46:17.899365 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:17.899312    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gs6zr" podStartSLOduration=2.246585965 podStartE2EDuration="3.89928114s" totalImagesPullingTime="1.652695175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:46:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:17.89878079 +0000 UTC m=+38.992409700" watchObservedRunningTime="2026-10-29 21:46:17.89928114 +0000 UTC m=+38.992910045"
Oct 29 21:46:18.622132 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:18.622110    2579 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:18.622231 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:18.622157    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:18.622504 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:18.622490    2579 scope.go:122] "RemoveContainer" containerID="b727bfa5392f31848f4d9bdd2173e4c65b562208ae7f10f517776dd67c5132f8"
Oct 29 21:46:18.622666 ip-10-0-129-146 kubenswrapper[2579]: E1029 21:46:18.622649    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jdps8_openshift-console-operator(f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8" podUID="f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600"
Oct 29 21:46:18.854429 ip-10-0-129-146 kubenswrapper[2579]: I1029 21:46:18.854348    2579 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 21:46:18.854518 ip-10-0-129-146 systemd[1]: Stopping Kubernetes Kubelet...
Oct 29 21:46:18.865667 ip-10-0-129-146 systemd[1]: kubelet.service: Deactivated successfully.
Oct 29 21:46:18.865829 ip-10-0-129-146 systemd[1]: Stopped Kubernetes Kubelet.
Oct 29 21:46:18.865875 ip-10-0-129-146 systemd[1]: kubelet.service: Consumed 2.823s CPU time.
Oct 29 21:46:18.877774 ip-10-0-129-146 systemd[1]: Starting Kubernetes Kubelet...
Oct 29 21:46:18.956640 ip-10-0-129-146 kubenswrapper[4962]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 21:46:18.956640 ip-10-0-129-146 kubenswrapper[4962]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 29 21:46:18.956973 ip-10-0-129-146 kubenswrapper[4962]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 21:46:18.956973 ip-10-0-129-146 kubenswrapper[4962]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 21:46:18.958751 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958684    4962 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:46:18.958751 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958752    4962 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:46:18.958819 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958756    4962 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:46:18.958819 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958759    4962 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:46:18.958819 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958762    4962 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:46:18.958819 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958765    4962 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:46:18.958819 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958767    4962 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:46:18.958819 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958771    4962 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:46:18.958819 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958774    4962 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:46:18.958819 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958777    4962 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:46:18.958819 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958779    4962 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:46:18.958819 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958784    4962 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:46:18.958819 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958789    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:46:18.958819 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958792    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:46:18.958819 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958794    4962 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:46:18.958819 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958797    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:46:18.958819 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958800    4962 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:46:18.958819 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958802    4962 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:46:18.958819 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958805    4962 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:46:18.958819 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958808    4962 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:46:18.958819 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958810    4962 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:46:18.959282 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958813    4962 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:46:18.959282 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958817    4962 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:46:18.959282 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958819    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:46:18.959282 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958822    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:46:18.959282 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958825    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:46:18.959282 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958828    4962 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:46:18.959282 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958831    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:46:18.959282 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958834    4962 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:46:18.959282 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958836    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:46:18.959282 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958839    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:46:18.959282 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958842    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:46:18.959282 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958844    4962 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:46:18.959282 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958847    4962 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:46:18.959282 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958850    4962 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:46:18.959282 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958853    4962 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:46:18.959282 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958856    4962 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:46:18.959282 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958858    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:46:18.959282 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958861    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:46:18.959282 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958864    4962 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:46:18.959755 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958866    4962 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:46:18.959755 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958869    4962 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:46:18.959755 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958871    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:46:18.959755 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958874    4962 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:46:18.959755 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958877    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:46:18.959755 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958879    4962 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:46:18.959755 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958882    4962 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:46:18.959755 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958885    4962 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:46:18.959755 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958888    4962 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:46:18.959755 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958891    4962 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:46:18.959755 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958894    4962 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:46:18.959755 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958896    4962 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:46:18.959755 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958899    4962 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:46:18.959755 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958902    4962 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:46:18.959755 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958904    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:46:18.959755 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958907    4962 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:46:18.959755 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958910    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:46:18.959755 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958912    4962 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:46:18.959755 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958915    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:46:18.959755 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958919    4962 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:46:18.960245 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958921    4962 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:46:18.960245 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958924    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:46:18.960245 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958929    4962 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:46:18.960245 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958933    4962 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:46:18.960245 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958936    4962 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:46:18.960245 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958939    4962 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:46:18.960245 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958942    4962 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:46:18.960245 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958945    4962 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:46:18.960245 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958947    4962 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:46:18.960245 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958950    4962 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:46:18.960245 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958953    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:46:18.960245 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958956    4962 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:46:18.960245 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958958    4962 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:46:18.960245 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958961    4962 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:46:18.960245 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958964    4962 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:46:18.960245 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958966    4962 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:46:18.960245 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958969    4962 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:46:18.960245 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958972    4962 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:46:18.960245 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958975    4962 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:46:18.960717 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958977    4962 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:46:18.960717 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958980    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:46:18.960717 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958982    4962 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:46:18.960717 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958985    4962 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:46:18.960717 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958988    4962 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:46:18.960717 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958991    4962 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:46:18.960717 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958993    4962 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:46:18.960717 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958996    4962 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:46:18.960717 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.958999    4962 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:46:18.960717 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959002    4962 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:46:18.960717 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959004    4962 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:46:18.960717 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959007    4962 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:46:18.960717 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959009    4962 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:46:18.960717 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959012    4962 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:46:18.960717 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959016    4962 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:46:18.960717 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959018    4962 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:46:18.960717 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959021    4962 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:46:18.960717 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959023    4962 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:46:18.960717 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959026    4962 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:46:18.960717 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959029    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:46:18.961218 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959031    4962 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:46:18.961218 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959034    4962 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:46:18.961218 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959036    4962 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:46:18.961218 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959039    4962 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:46:18.961218 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959041    4962 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:46:18.961218 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959044    4962 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:46:18.961218 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959047    4962 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:46:18.961218 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959789    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:46:18.961218 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959799    4962 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:46:18.961218 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959803    4962 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:46:18.961218 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959807    4962 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:46:18.961218 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959810    4962 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:46:18.961218 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959813    4962 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:46:18.961218 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959816    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:46:18.961218 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959819    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:46:18.961218 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959822    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:46:18.961218 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959824    4962 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:46:18.961218 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959827    4962 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:46:18.961218 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959831    4962 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:46:18.961744 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959836    4962 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:46:18.961744 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959839    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:46:18.961744 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959842    4962 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:46:18.961744 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959845    4962 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:46:18.961744 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959847    4962 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:46:18.961744 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959850    4962 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:46:18.961744 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959852    4962 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:46:18.961744 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959855    4962 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:46:18.961744 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959858    4962 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:46:18.961744 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959861    4962 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:46:18.961744 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959863    4962 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:46:18.961744 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959866    4962 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:46:18.961744 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959869    4962 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:46:18.961744 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959872    4962 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:46:18.961744 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959875    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:46:18.961744 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959877    4962 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:46:18.961744 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959880    4962 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:46:18.961744 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959883    4962 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:46:18.961744 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959885    4962 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:46:18.962223 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959888    4962 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:46:18.962223 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959891    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:46:18.962223 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959894    4962 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:46:18.962223 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959897    4962 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:46:18.962223 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959899    4962 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:46:18.962223 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959902    4962 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:46:18.962223 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959904    4962 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:46:18.962223 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959907    4962 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:46:18.962223 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959910    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:46:18.962223 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959912    4962 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:46:18.962223 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959915    4962 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:46:18.962223 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959918    4962 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:46:18.962223 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959921    4962 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:46:18.962223 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959924    4962 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:46:18.962223 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959926    4962 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:46:18.962223 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959929    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:46:18.962223 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959932    4962 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:46:18.962223 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959934    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:46:18.962223 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959937    4962 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:46:18.962696 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959939    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:46:18.962696 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959942    4962 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:46:18.962696 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959945    4962 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:46:18.962696 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959947    4962 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:46:18.962696 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959950    4962 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:46:18.962696 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959953    4962 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:46:18.962696 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959956    4962 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:46:18.962696 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959959    4962 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:46:18.962696 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959971    4962 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:46:18.962696 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959974    4962 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:46:18.962696 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959977    4962 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:46:18.962696 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959980    4962 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:46:18.962696 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959982    4962 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:46:18.962696 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959985    4962 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:46:18.962696 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959988    4962 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:46:18.962696 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959990    4962 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:46:18.962696 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959993    4962 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:46:18.962696 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959996    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:46:18.962696 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.959998    4962 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:46:18.963163 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960001    4962 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:46:18.963163 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960003    4962 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:46:18.963163 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960006    4962 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:46:18.963163 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960009    4962 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:46:18.963163 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960011    4962 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:46:18.963163 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960014    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:46:18.963163 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960016    4962 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:46:18.963163 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960019    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:46:18.963163 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960021    4962 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:46:18.963163 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960024    4962 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:46:18.963163 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960027    4962 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:46:18.963163 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960029    4962 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:46:18.963163 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960032    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:46:18.963163 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960034    4962 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:46:18.963163 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960037    4962 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:46:18.963163 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960040    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:46:18.963163 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960043    4962 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:46:18.963163 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960046    4962 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:46:18.963163 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960049    4962 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:46:18.963163 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960052    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:46:18.963679 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960054    4962 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:46:18.963679 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960057    4962 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:46:18.963679 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960065    4962 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:46:18.963679 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960068    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:46:18.963679 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960071    4962 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:46:18.963679 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960074    4962 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:46:18.963679 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960076    4962 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:46:18.963679 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960079    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:46:18.963679 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960082    4962 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:46:18.963679 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960085    4962 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:46:18.963679 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960088    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:46:18.963679 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960090    4962 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:46:18.963679 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960093    4962 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:46:18.963679 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960096    4962 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:46:18.963679 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960098    4962 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:46:18.963679 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960101    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:46:18.963679 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960104    4962 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:46:18.963679 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960620    4962 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:46:18.963679 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960625    4962 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:46:18.963679 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960628    4962 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:46:18.964159 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960631    4962 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:46:18.964159 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960634    4962 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:46:18.964159 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960636    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:46:18.964159 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960639    4962 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:46:18.964159 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960642    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:46:18.964159 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960645    4962 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:46:18.964159 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960648    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:46:18.964159 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960651    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:46:18.964159 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960653    4962 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:46:18.964159 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960656    4962 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:46:18.964159 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960659    4962 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:46:18.964159 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960662    4962 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:46:18.964159 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960664    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:46:18.964159 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960667    4962 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:46:18.964159 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960670    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:46:18.964159 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960673    4962 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:46:18.964159 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960675    4962 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:46:18.964159 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960678    4962 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:46:18.964159 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960681    4962 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:46:18.964159 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960683    4962 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:46:18.964712 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960686    4962 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:46:18.964712 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960690    4962 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:46:18.964712 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960694    4962 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:46:18.964712 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960697    4962 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:46:18.964712 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960700    4962 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:46:18.964712 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960702    4962 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:46:18.964712 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960705    4962 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:46:18.964712 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960708    4962 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:46:18.964712 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960710    4962 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:46:18.964712 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960713    4962 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:46:18.964712 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960716    4962 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:46:18.964712 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960719    4962 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:46:18.964712 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960721    4962 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:46:18.964712 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960724    4962 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:46:18.964712 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960727    4962 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:46:18.964712 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960730    4962 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:46:18.964712 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960733    4962 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:46:18.964712 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960737    4962 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:46:18.964712 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960740    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:46:18.965489 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960743    4962 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:46:18.965489 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960746    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:46:18.965489 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960749    4962 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:46:18.965489 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960751    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:46:18.965489 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960754    4962 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:46:18.965489 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960756    4962 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:46:18.965489 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960759    4962 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:46:18.965489 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960761    4962 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:46:18.965489 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960765    4962 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:46:18.965489 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960768    4962 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:46:18.965489 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960770    4962 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:46:18.965489 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960773    4962 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:46:18.965489 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960775    4962 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:46:18.965489 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960778    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:46:18.965489 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960781    4962 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:46:18.965489 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960787    4962 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:46:18.965489 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960790    4962 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:46:18.965489 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960792    4962 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:46:18.965489 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960795    4962 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:46:18.965489 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960797    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:46:18.965989 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960800    4962 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:46:18.965989 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960803    4962 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:46:18.965989 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960806    4962 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:46:18.965989 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960808    4962 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:46:18.965989 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960820    4962 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:46:18.965989 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960823    4962 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:46:18.965989 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960826    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:46:18.965989 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960828    4962 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:46:18.965989 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960831    4962 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:46:18.965989 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960834    4962 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:46:18.965989 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960837    4962 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:46:18.965989 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960840    4962 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:46:18.965989 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960842    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:46:18.965989 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960845    4962 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:46:18.965989 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960847    4962 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:46:18.965989 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960850    4962 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:46:18.965989 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960853    4962 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:46:18.965989 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960856    4962 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:46:18.965989 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960859    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:46:18.966493 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960862    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:46:18.966493 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960864    4962 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:46:18.966493 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960867    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:46:18.966493 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960870    4962 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:46:18.966493 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960872    4962 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:46:18.966493 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960875    4962 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:46:18.966493 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960878    4962 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:46:18.966493 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960880    4962 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:46:18.966493 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960884    4962 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:46:18.966493 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960887    4962 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:46:18.966493 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960890    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:46:18.966493 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960893    4962 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:46:18.966493 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960895    4962 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:46:18.966493 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960898    4962 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:46:18.966493 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960900    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:46:18.966493 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960903    4962 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:46:18.966493 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960906    4962 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:46:18.966493 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960908    4962 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:46:18.966493 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960912    4962 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:46:18.966493 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960915    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:46:18.966977 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960917    4962 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:46:18.966977 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960920    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:46:18.966977 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960922    4962 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:46:18.966977 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960926    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:46:18.966977 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.960929    4962 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:46:18.966977 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961018    4962 flags.go:64] FLAG: --address="0.0.0.0"
Oct 29 21:46:18.966977 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961026    4962 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 29 21:46:18.966977 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961032    4962 flags.go:64] FLAG: --anonymous-auth="true"
Oct 29 21:46:18.966977 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961036    4962 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 29 21:46:18.966977 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961041    4962 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 29 21:46:18.966977 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961044    4962 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 29 21:46:18.966977 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961048    4962 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 29 21:46:18.966977 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961053    4962 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 29 21:46:18.966977 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961056    4962 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 29 21:46:18.966977 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961059    4962 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 29 21:46:18.966977 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961063    4962 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 29 21:46:18.966977 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961066    4962 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 29 21:46:18.966977 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961069    4962 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 29 21:46:18.966977 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961072    4962 flags.go:64] FLAG: --cgroup-root=""
Oct 29 21:46:18.966977 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961075    4962 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 29 21:46:18.966977 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961080    4962 flags.go:64] FLAG: --client-ca-file=""
Oct 29 21:46:18.966977 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961083    4962 flags.go:64] FLAG: --cloud-provider="external"
Oct 29 21:46:18.966977 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961086    4962 flags.go:64] FLAG: --cluster-dns="[]"
Oct 29 21:46:18.967594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961090    4962 flags.go:64] FLAG: --cluster-domain=""
Oct 29 21:46:18.967594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961093    4962 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 29 21:46:18.967594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961096    4962 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 29 21:46:18.967594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961099    4962 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 29 21:46:18.967594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961104    4962 flags.go:64] FLAG: --container-log-max-files="5"
Oct 29 21:46:18.967594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961109    4962 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 29 21:46:18.967594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961114    4962 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 29 21:46:18.967594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961119    4962 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 29 21:46:18.967594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961122    4962 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 29 21:46:18.967594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961126    4962 flags.go:64] FLAG: --contention-profiling="false"
Oct 29 21:46:18.967594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961129    4962 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 29 21:46:18.967594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961132    4962 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 29 21:46:18.967594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961137    4962 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 29 21:46:18.967594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961140    4962 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 29 21:46:18.967594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961144    4962 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 29 21:46:18.967594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961147    4962 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 29 21:46:18.967594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961150    4962 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 29 21:46:18.967594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961153    4962 flags.go:64] FLAG: --enable-load-reader="false"
Oct 29 21:46:18.967594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961156    4962 flags.go:64] FLAG: --enable-server="true"
Oct 29 21:46:18.967594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961159    4962 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 29 21:46:18.967594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961164    4962 flags.go:64] FLAG: --event-burst="100"
Oct 29 21:46:18.967594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961167    4962 flags.go:64] FLAG: --event-qps="50"
Oct 29 21:46:18.967594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961170    4962 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 29 21:46:18.967594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961173    4962 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 29 21:46:18.968173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961176    4962 flags.go:64] FLAG: --eviction-hard=""
Oct 29 21:46:18.968173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961180    4962 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 29 21:46:18.968173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961183    4962 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 29 21:46:18.968173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961187    4962 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 29 21:46:18.968173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961190    4962 flags.go:64] FLAG: --eviction-soft=""
Oct 29 21:46:18.968173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961195    4962 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 29 21:46:18.968173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961198    4962 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 29 21:46:18.968173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961201    4962 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 29 21:46:18.968173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961204    4962 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 29 21:46:18.968173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961207    4962 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 29 21:46:18.968173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961210    4962 flags.go:64] FLAG: --fail-swap-on="true"
Oct 29 21:46:18.968173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961213    4962 flags.go:64] FLAG: --feature-gates=""
Oct 29 21:46:18.968173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961217    4962 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 29 21:46:18.968173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961220    4962 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 29 21:46:18.968173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961223    4962 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 29 21:46:18.968173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961227    4962 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 29 21:46:18.968173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961230    4962 flags.go:64] FLAG: --healthz-port="10248"
Oct 29 21:46:18.968173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961233    4962 flags.go:64] FLAG: --help="false"
Oct 29 21:46:18.968173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961236    4962 flags.go:64] FLAG: --hostname-override="ip-10-0-129-146.ec2.internal"
Oct 29 21:46:18.968173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961239    4962 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 29 21:46:18.968173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961246    4962 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 29 21:46:18.968173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961249    4962 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 29 21:46:18.968173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961253    4962 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 29 21:46:18.968173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961256    4962 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961259    4962 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961262    4962 flags.go:64] FLAG: --image-service-endpoint=""
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961265    4962 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961268    4962 flags.go:64] FLAG: --kube-api-burst="100"
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961271    4962 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961274    4962 flags.go:64] FLAG: --kube-api-qps="50"
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961277    4962 flags.go:64] FLAG: --kube-reserved=""
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961280    4962 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961296    4962 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961302    4962 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961307    4962 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961310    4962 flags.go:64] FLAG: --lock-file=""
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961313    4962 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961319    4962 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961322    4962 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961327    4962 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961330    4962 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961334    4962 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961337    4962 flags.go:64] FLAG: --logging-format="text"
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961340    4962 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961343    4962 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961346    4962 flags.go:64] FLAG: --manifest-url=""
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961349    4962 flags.go:64] FLAG: --manifest-url-header=""
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961354    4962 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 29 21:46:18.968890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961357    4962 flags.go:64] FLAG: --max-open-files="1000000"
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961361    4962 flags.go:64] FLAG: --max-pods="110"
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961364    4962 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961368    4962 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961372    4962 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961376    4962 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961379    4962 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961382    4962 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961384    4962 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961391    4962 flags.go:64] FLAG: --node-status-max-images="50"
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961394    4962 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961398    4962 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961401    4962 flags.go:64] FLAG: --pod-cidr=""
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961404    4962 flags.go:64] FLAG: --pod-manifest-path=""
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961406    4962 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961410    4962 flags.go:64] FLAG: --pods-per-core="0"
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961413    4962 flags.go:64] FLAG: --port="10250"
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961416    4962 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961419    4962 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-038c5f85302bf0b24"
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961423    4962 flags.go:64] FLAG: --qos-reserved=""
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961426    4962 flags.go:64] FLAG: --read-only-port="10255"
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961430    4962 flags.go:64] FLAG: --register-node="true"
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961433    4962 flags.go:64] FLAG: --register-with-taints=""
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961437    4962 flags.go:64] FLAG: --registry-burst="10"
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961440    4962 flags.go:64] FLAG: --registry-qps="5"
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961443    4962 flags.go:64] FLAG: --reserved-cpus=""
Oct 29 21:46:18.969504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961446    4962 flags.go:64] FLAG: --reserved-memory=""
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961450    4962 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961453    4962 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961456    4962 flags.go:64] FLAG: --rotate-certificates="false"
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961459    4962 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961462    4962 flags.go:64] FLAG: --runonce="false"
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961465    4962 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961468    4962 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961471    4962 flags.go:64] FLAG: --seccomp-default="false"
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961474    4962 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961478    4962 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961482    4962 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961485    4962 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961488    4962 flags.go:64] FLAG: --storage-driver-password="root"
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961491    4962 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961493    4962 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961497    4962 flags.go:64] FLAG: --storage-driver-user="root"
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961500    4962 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961503    4962 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961506    4962 flags.go:64] FLAG: --system-cgroups=""
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961509    4962 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961514    4962 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961517    4962 flags.go:64] FLAG: --tls-cert-file=""
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961520    4962 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961524    4962 flags.go:64] FLAG: --tls-min-version=""
Oct 29 21:46:18.970144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961527    4962 flags.go:64] FLAG: --tls-private-key-file=""
Oct 29 21:46:18.970751 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961531    4962 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 29 21:46:18.970751 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961535    4962 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 29 21:46:18.970751 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961538    4962 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 29 21:46:18.970751 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961541    4962 flags.go:64] FLAG: --v="2"
Oct 29 21:46:18.970751 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961545    4962 flags.go:64] FLAG: --version="false"
Oct 29 21:46:18.970751 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961549    4962 flags.go:64] FLAG: --vmodule=""
Oct 29 21:46:18.970751 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961554    4962 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 29 21:46:18.970751 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.961557    4962 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 29 21:46:18.970751 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961701    4962 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:46:18.970751 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961705    4962 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:46:18.970751 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961708    4962 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:46:18.970751 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961711    4962 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:46:18.970751 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961714    4962 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:46:18.970751 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961717    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:46:18.970751 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961719    4962 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:46:18.970751 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961722    4962 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:46:18.970751 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961727    4962 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:46:18.970751 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961729    4962 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:46:18.970751 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961732    4962 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:46:18.970751 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961735    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:46:18.970751 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961738    4962 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:46:18.971240 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961741    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:46:18.971240 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961743    4962 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:46:18.971240 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961746    4962 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:46:18.971240 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961749    4962 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:46:18.971240 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961752    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:46:18.971240 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961755    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:46:18.971240 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961757    4962 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:46:18.971240 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961760    4962 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:46:18.971240 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961763    4962 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:46:18.971240 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961765    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:46:18.971240 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961768    4962 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:46:18.971240 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961771    4962 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:46:18.971240 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961775    4962 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:46:18.971240 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961778    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:46:18.971240 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961780    4962 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:46:18.971240 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961783    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:46:18.971240 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961785    4962 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:46:18.971240 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961788    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:46:18.971240 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961791    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:46:18.971240 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961793    4962 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:46:18.971885 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961796    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:46:18.971885 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961798    4962 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:46:18.971885 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961801    4962 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:46:18.971885 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961804    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:46:18.971885 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961806    4962 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:46:18.971885 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961809    4962 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:46:18.971885 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961811    4962 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:46:18.971885 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961815    4962 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:46:18.971885 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961818    4962 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:46:18.971885 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961821    4962 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:46:18.971885 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961825    4962 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:46:18.971885 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961829    4962 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:46:18.971885 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961832    4962 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:46:18.971885 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961835    4962 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:46:18.971885 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961838    4962 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:46:18.971885 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961841    4962 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:46:18.971885 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961843    4962 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:46:18.971885 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961846    4962 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:46:18.971885 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961848    4962 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:46:18.972437 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961851    4962 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:46:18.972437 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961854    4962 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:46:18.972437 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961856    4962 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:46:18.972437 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961859    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:46:18.972437 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961862    4962 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:46:18.972437 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961865    4962 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:46:18.972437 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961868    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:46:18.972437 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961870    4962 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:46:18.972437 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961873    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:46:18.972437 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961876    4962 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:46:18.972437 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961878    4962 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:46:18.972437 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961881    4962 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:46:18.972437 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961884    4962 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:46:18.972437 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961886    4962 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:46:18.972437 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961889    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:46:18.972437 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961891    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:46:18.972437 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961894    4962 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:46:18.972437 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961896    4962 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:46:18.972437 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961899    4962 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:46:18.972437 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961902    4962 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:46:18.972960 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961905    4962 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:46:18.972960 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961908    4962 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:46:18.972960 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961910    4962 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:46:18.972960 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961913    4962 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:46:18.972960 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961916    4962 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:46:18.972960 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961918    4962 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:46:18.972960 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961921    4962 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:46:18.972960 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961923    4962 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:46:18.972960 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961926    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:46:18.972960 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961929    4962 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:46:18.972960 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961932    4962 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:46:18.972960 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961936    4962 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:46:18.972960 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961939    4962 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:46:18.972960 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961942    4962 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:46:18.972960 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961945    4962 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:46:18.972960 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961947    4962 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:46:18.972960 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961950    4962 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:46:18.972960 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961952    4962 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:46:18.972960 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961955    4962 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:46:18.972960 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961958    4962 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:46:18.973448 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961961    4962 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:46:18.973448 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961963    4962 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:46:18.973448 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961966    4962 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:46:18.973448 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961968    4962 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:46:18.973448 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961971    4962 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:46:18.973448 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961974    4962 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:46:18.973448 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961976    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:46:18.973448 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961979    4962 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:46:18.973448 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961981    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:46:18.973448 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961984    4962 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:46:18.973448 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961986    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:46:18.973448 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961989    4962 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:46:18.973448 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961993    4962 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:46:18.973448 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.961995    4962 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:46:18.973448 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.962020    4962 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 21:46:18.973879 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962067    4962 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:46:18.973879 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962072    4962 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:46:18.973879 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962075    4962 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:46:18.973879 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962078    4962 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:46:18.973879 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962080    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:46:18.973879 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962083    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:46:18.973879 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962086    4962 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:46:18.973879 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962089    4962 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:46:18.973879 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962091    4962 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:46:18.973879 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962094    4962 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:46:18.973879 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962096    4962 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:46:18.973879 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962099    4962 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:46:18.973879 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962101    4962 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:46:18.973879 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962104    4962 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:46:18.973879 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962106    4962 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:46:18.973879 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962109    4962 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:46:18.973879 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962111    4962 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:46:18.973879 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962114    4962 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:46:18.973879 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962117    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:46:18.973879 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962119    4962 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:46:18.974543 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962123    4962 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:46:18.974543 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962127    4962 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:46:18.974543 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962130    4962 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:46:18.974543 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962133    4962 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:46:18.974543 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962135    4962 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:46:18.974543 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962138    4962 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:46:18.974543 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962140    4962 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:46:18.974543 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962143    4962 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:46:18.974543 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962146    4962 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:46:18.974543 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962149    4962 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:46:18.974543 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962152    4962 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:46:18.974543 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962155    4962 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:46:18.974543 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962158    4962 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:46:18.974543 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962161    4962 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:46:18.974543 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962164    4962 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:46:18.974543 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962167    4962 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:46:18.974543 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962169    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:46:18.974543 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962172    4962 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:46:18.974543 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962174    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:46:18.975021 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962177    4962 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:46:18.975021 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962180    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:46:18.975021 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962183    4962 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:46:18.975021 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962185    4962 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:46:18.975021 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962188    4962 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:46:18.975021 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962190    4962 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:46:18.975021 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962193    4962 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:46:18.975021 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962196    4962 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:46:18.975021 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962198    4962 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:46:18.975021 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962201    4962 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:46:18.975021 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962204    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:46:18.975021 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962207    4962 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:46:18.975021 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962209    4962 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:46:18.975021 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962212    4962 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:46:18.975021 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962215    4962 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:46:18.975021 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962217    4962 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:46:18.975021 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962220    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:46:18.975021 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962223    4962 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:46:18.975021 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962226    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:46:18.975561 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962228    4962 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:46:18.975561 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962231    4962 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:46:18.975561 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962234    4962 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:46:18.975561 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962238    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:46:18.975561 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962241    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:46:18.975561 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962244    4962 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:46:18.975561 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962248    4962 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:46:18.975561 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962250    4962 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:46:18.975561 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962253    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:46:18.975561 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962256    4962 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:46:18.975561 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962259    4962 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:46:18.975561 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962262    4962 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:46:18.975561 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962264    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:46:18.975561 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962267    4962 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:46:18.975561 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962270    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:46:18.975561 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962272    4962 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:46:18.975561 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962275    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:46:18.975561 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962277    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:46:18.975561 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962280    4962 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:46:18.975561 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962296    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:46:18.976034 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962300    4962 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:46:18.976034 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962303    4962 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:46:18.976034 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962306    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:46:18.976034 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962309    4962 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:46:18.976034 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962311    4962 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:46:18.976034 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962315    4962 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:46:18.976034 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962317    4962 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:46:18.976034 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962320    4962 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:46:18.976034 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962323    4962 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:46:18.976034 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962327    4962 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:46:18.976034 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962331    4962 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:46:18.976034 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962335    4962 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:46:18.976034 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962338    4962 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:46:18.976034 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962341    4962 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:46:18.976034 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962344    4962 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:46:18.976034 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962348    4962 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:46:18.976034 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962351    4962 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:46:18.976034 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962354    4962 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:46:18.976034 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962358    4962 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:46:18.976034 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962360    4962 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:46:18.976540 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962363    4962 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:46:18.976540 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962365    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:46:18.976540 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962368    4962 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:46:18.976540 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962371    4962 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:46:18.976540 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962374    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:46:18.976540 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962376    4962 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:46:18.976540 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962381    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:46:18.976540 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.962383    4962 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:46:18.976540 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.966836    4962 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 29 21:46:18.976540 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.966853    4962 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 29 21:46:18.976540 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.966871    4962 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 29 21:46:18.976540 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.966876    4962 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 29 21:46:18.976540 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.966935    4962 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:46:18.976540 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.966941    4962 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:46:18.976540 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.966946    4962 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:46:18.976540 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.966950    4962 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:46:18.976540 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.966953    4962 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:46:18.976540 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.966956    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:46:18.976540 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.966959    4962 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:46:18.977015 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.966962    4962 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:46:18.977015 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.966965    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:46:18.977015 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.966967    4962 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:46:18.977015 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.966971    4962 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:46:18.977015 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.966973    4962 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:46:18.977015 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.966976    4962 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:46:18.977015 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.966979    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:46:18.977015 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.966982    4962 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:46:18.977015 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.966986    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:46:18.977015 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.966989    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:46:18.977015 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.966992    4962 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:46:18.977015 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.966994    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:46:18.977015 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.966997    4962 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:46:18.977015 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967000    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:46:18.977015 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967002    4962 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:46:18.977015 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967005    4962 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:46:18.977015 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967008    4962 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:46:18.977015 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967011    4962 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:46:18.977015 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967013    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:46:18.977015 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967016    4962 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:46:18.977662 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967019    4962 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:46:18.977662 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967022    4962 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:46:18.977662 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967025    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:46:18.977662 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967028    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:46:18.977662 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967031    4962 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:46:18.977662 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967033    4962 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:46:18.977662 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967036    4962 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:46:18.977662 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967039    4962 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:46:18.977662 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967041    4962 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:46:18.977662 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967044    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:46:18.977662 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967047    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:46:18.977662 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967049    4962 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:46:18.977662 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967052    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:46:18.977662 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967055    4962 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:46:18.977662 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967057    4962 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:46:18.977662 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967060    4962 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:46:18.977662 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967062    4962 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:46:18.977662 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967065    4962 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:46:18.977662 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967068    4962 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:46:18.978438 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967071    4962 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:46:18.978438 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967075    4962 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:46:18.978438 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967078    4962 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:46:18.978438 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967080    4962 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:46:18.978438 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967083    4962 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:46:18.978438 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967086    4962 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:46:18.978438 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967088    4962 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:46:18.978438 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967091    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:46:18.978438 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967094    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:46:18.978438 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967096    4962 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:46:18.978438 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967099    4962 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:46:18.978438 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967102    4962 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:46:18.978438 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967104    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:46:18.978438 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967107    4962 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:46:18.978438 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967110    4962 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:46:18.978438 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967112    4962 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:46:18.978438 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967115    4962 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:46:18.978438 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967118    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:46:18.978438 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967120    4962 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:46:18.979130 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967123    4962 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:46:18.979130 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967126    4962 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:46:18.979130 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967128    4962 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:46:18.979130 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967131    4962 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:46:18.979130 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967133    4962 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:46:18.979130 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967136    4962 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:46:18.979130 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967138    4962 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:46:18.979130 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967141    4962 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:46:18.979130 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967144    4962 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:46:18.979130 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967147    4962 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:46:18.979130 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967149    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:46:18.979130 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967152    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:46:18.979130 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967155    4962 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:46:18.979130 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967158    4962 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:46:18.979130 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967161    4962 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:46:18.979130 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967165    4962 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:46:18.979130 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967169    4962 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:46:18.979130 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967172    4962 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:46:18.979130 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967175    4962 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:46:18.979967 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967178    4962 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:46:18.979967 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967181    4962 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:46:18.979967 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967183    4962 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:46:18.979967 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967186    4962 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:46:18.979967 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967189    4962 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:46:18.979967 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967191    4962 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:46:18.979967 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967194    4962 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:46:18.979967 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967196    4962 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:46:18.979967 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967199    4962 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:46:18.979967 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967202    4962 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:46:18.979967 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967204    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:46:18.979967 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967207    4962 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:46:18.979967 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967210    4962 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:46:18.979967 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967213    4962 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:46:18.979967 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967215    4962 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:46:18.979967 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967218    4962 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:46:18.979967 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967221    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:46:18.979967 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967223    4962 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:46:18.979967 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967226    4962 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:46:18.979967 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967228    4962 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:46:18.980808 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967231    4962 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:46:18.980808 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967234    4962 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:46:18.980808 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.967262    4962 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 21:46:18.980808 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967406    4962 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:46:18.980808 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967412    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:46:18.980808 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967415    4962 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:46:18.980808 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967418    4962 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:46:18.980808 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967420    4962 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:46:18.980808 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967424    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:46:18.980808 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967426    4962 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:46:18.980808 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967429    4962 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:46:18.980808 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967432    4962 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:46:18.980808 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967434    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:46:18.980808 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967437    4962 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:46:18.980808 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967440    4962 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:46:18.980808 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967442    4962 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:46:18.981593 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967446    4962 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:46:18.981593 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967448    4962 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:46:18.981593 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967451    4962 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:46:18.981593 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967454    4962 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:46:18.981593 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967456    4962 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:46:18.981593 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967459    4962 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:46:18.981593 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967461    4962 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:46:18.981593 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967464    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:46:18.981593 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967467    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:46:18.981593 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967470    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:46:18.981593 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967474    4962 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:46:18.981593 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967478    4962 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:46:18.981593 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967481    4962 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:46:18.981593 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967484    4962 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:46:18.981593 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967487    4962 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:46:18.981593 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967490    4962 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:46:18.981593 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967492    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:46:18.981593 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967495    4962 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:46:18.981593 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967502    4962 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:46:18.982488 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967505    4962 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:46:18.982488 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967509    4962 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:46:18.982488 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967512    4962 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:46:18.982488 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967515    4962 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:46:18.982488 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967518    4962 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:46:18.982488 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967520    4962 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:46:18.982488 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967524    4962 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:46:18.982488 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967526    4962 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:46:18.982488 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967529    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:46:18.982488 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967532    4962 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:46:18.982488 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967534    4962 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:46:18.982488 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967537    4962 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:46:18.982488 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967540    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:46:18.982488 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967543    4962 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:46:18.982488 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967545    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:46:18.982488 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967548    4962 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:46:18.982488 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967551    4962 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:46:18.982488 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967553    4962 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:46:18.982488 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967556    4962 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:46:18.982488 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967559    4962 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:46:18.983390 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967561    4962 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:46:18.983390 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967564    4962 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:46:18.983390 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967566    4962 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:46:18.983390 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967569    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:46:18.983390 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967572    4962 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:46:18.983390 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967574    4962 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:46:18.983390 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967577    4962 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:46:18.983390 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967579    4962 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:46:18.983390 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967582    4962 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:46:18.983390 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967585    4962 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:46:18.983390 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967588    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:46:18.983390 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967592    4962 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:46:18.983390 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967594    4962 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:46:18.983390 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967597    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:46:18.983390 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967600    4962 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:46:18.983390 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967603    4962 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:46:18.983390 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967606    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:46:18.983390 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967608    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:46:18.983390 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967611    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:46:18.984514 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967614    4962 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:46:18.984514 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967617    4962 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:46:18.984514 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967620    4962 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:46:18.984514 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967622    4962 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:46:18.984514 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967625    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:46:18.984514 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967628    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:46:18.984514 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967630    4962 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:46:18.984514 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967633    4962 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:46:18.984514 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967636    4962 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:46:18.984514 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967639    4962 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:46:18.984514 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967642    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:46:18.984514 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967644    4962 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:46:18.984514 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967647    4962 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:46:18.984514 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967649    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:46:18.984514 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967652    4962 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:46:18.984514 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967655    4962 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:46:18.984514 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967657    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:46:18.984514 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967660    4962 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:46:18.984514 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967663    4962 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:46:18.985411 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967665    4962 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:46:18.985411 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967668    4962 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:46:18.985411 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967671    4962 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:46:18.985411 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967673    4962 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:46:18.985411 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967676    4962 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:46:18.985411 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967679    4962 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:46:18.985411 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967682    4962 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:46:18.985411 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967685    4962 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:46:18.985411 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967687    4962 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:46:18.985411 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967690    4962 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:46:18.985411 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967693    4962 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:46:18.985411 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967695    4962 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:46:18.985411 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967698    4962 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:46:18.985411 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967701    4962 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:46:18.985411 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967704    4962 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:46:18.985411 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967707    4962 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:46:18.986146 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.967731    4962 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 21:46:18.986146 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967763    4962 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 21:46:18.986146 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967767    4962 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 21:46:18.986146 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967770    4962 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 21:46:18.986146 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967773    4962 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 21:46:18.986146 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967775    4962 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 21:46:18.986146 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967778    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 21:46:18.986146 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967781    4962 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 21:46:18.986146 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967784    4962 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 21:46:18.986146 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967786    4962 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 21:46:18.986146 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967789    4962 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 21:46:18.986146 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967792    4962 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 21:46:18.986146 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967795    4962 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 21:46:18.986146 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967798    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 21:46:18.986146 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967800    4962 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 21:46:18.986146 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967803    4962 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 21:46:18.986831 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967806    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 21:46:18.986831 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967809    4962 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 21:46:18.986831 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967811    4962 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 21:46:18.986831 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967814    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 21:46:18.986831 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967816    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 21:46:18.986831 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967819    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 21:46:18.986831 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967823    4962 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 21:46:18.986831 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967826    4962 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 21:46:18.986831 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967829    4962 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 21:46:18.986831 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967831    4962 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 21:46:18.986831 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967834    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 21:46:18.986831 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967837    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 21:46:18.986831 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967840    4962 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 21:46:18.986831 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967842    4962 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 21:46:18.986831 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967845    4962 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 21:46:18.986831 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967848    4962 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 21:46:18.986831 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967851    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 21:46:18.986831 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967854    4962 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 21:46:18.986831 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967856    4962 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 21:46:18.987668 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967859    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 21:46:18.987668 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967861    4962 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 21:46:18.987668 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967864    4962 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 21:46:18.987668 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967866    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 21:46:18.987668 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967869    4962 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 21:46:18.987668 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967872    4962 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 21:46:18.987668 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967874    4962 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 21:46:18.987668 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967877    4962 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 21:46:18.987668 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967879    4962 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 21:46:18.987668 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967884    4962 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 21:46:18.987668 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967886    4962 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 21:46:18.987668 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967889    4962 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 21:46:18.987668 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967891    4962 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 21:46:18.987668 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967894    4962 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 21:46:18.987668 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967897    4962 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 21:46:18.987668 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967899    4962 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 21:46:18.987668 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967902    4962 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 21:46:18.987668 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967904    4962 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 21:46:18.987668 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967907    4962 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 21:46:18.988139 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967911    4962 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 21:46:18.988139 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967914    4962 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 21:46:18.988139 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967917    4962 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 21:46:18.988139 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967919    4962 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 21:46:18.988139 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967922    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 21:46:18.988139 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967925    4962 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 21:46:18.988139 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967927    4962 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 21:46:18.988139 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967930    4962 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 21:46:18.988139 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967933    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 21:46:18.988139 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967935    4962 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 21:46:18.988139 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967938    4962 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 21:46:18.988139 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967941    4962 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 21:46:18.988139 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967944    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 21:46:18.988139 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967946    4962 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 21:46:18.988139 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967949    4962 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 21:46:18.988139 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967951    4962 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 21:46:18.988139 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967954    4962 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 21:46:18.988139 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967957    4962 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 21:46:18.988139 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967960    4962 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 21:46:18.988139 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967962    4962 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 21:46:18.989012 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967965    4962 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 21:46:18.989012 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967968    4962 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 21:46:18.989012 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967972    4962 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 21:46:18.989012 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967974    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 21:46:18.989012 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967977    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 21:46:18.989012 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967981    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 21:46:18.989012 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967984    4962 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 21:46:18.989012 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967989    4962 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 21:46:18.989012 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967992    4962 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 21:46:18.989012 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967995    4962 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 21:46:18.989012 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.967997    4962 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 21:46:18.989012 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.968000    4962 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 21:46:18.989012 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.968004    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 21:46:18.989012 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.968007    4962 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 21:46:18.989012 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.968011    4962 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 21:46:18.989012 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.968015    4962 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 21:46:18.989012 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.968018    4962 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 21:46:18.989012 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.968020    4962 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 21:46:18.989012 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.968023    4962 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 21:46:18.989862 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.968026    4962 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 21:46:18.989862 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.968028    4962 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 21:46:18.989862 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.968031    4962 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 21:46:18.989862 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.968034    4962 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 21:46:18.989862 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.968037    4962 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 21:46:18.989862 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.968040    4962 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 21:46:18.989862 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.968042    4962 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 21:46:18.989862 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.968045    4962 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 21:46:18.989862 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.968048    4962 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 21:46:18.989862 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.968050    4962 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 21:46:18.989862 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.968053    4962 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 21:46:18.989862 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.968056    4962 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 21:46:18.989862 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.968058    4962 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 21:46:18.989862 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:18.968062    4962 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 21:46:18.989862 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.968119    4962 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 29 21:46:18.989862 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.969038    4962 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 29 21:46:18.989862 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.969090    4962 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 29 21:46:18.989862 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.969271    4962 server.go:1014] "Starting client certificate rotation"
Oct 29 21:46:18.989862 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.969385    4962 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 21:46:18.990706 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.969438    4962 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 21:40:39 +0000 UTC" deadline="2028-07-19 20:27:24.236772426 +0000 UTC"
Oct 29 21:46:18.990706 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.969476    4962 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15094h41m5.267298922s"
Oct 29 21:46:18.990706 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.969714    4962 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 21:46:18.990706 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.970749    4962 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 21:46:18.990706 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.972009    4962 log.go:25] "Validated CRI v1 runtime API"
Oct 29 21:46:18.990706 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.973596    4962 log.go:25] "Validated CRI v1 image API"
Oct 29 21:46:18.990706 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.974331    4962 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 29 21:46:18.990706 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.976051    4962 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c05f9a57-5ec0-4474-89ca-a39690efa53e:/dev/nvme0n1p3 df845650-e347-4498-a9f0-d5dc0ae2439a:/dev/nvme0n1p4]
Oct 29 21:46:18.991116 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.976069    4962 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/056b9baf976e19e2257192d2af06ff59d4fab0e77ec5e24badf9543b14134834/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/056b9baf976e19e2257192d2af06ff59d4fab0e77ec5e24badf9543b14134834/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/188b2721cbc4bb7917d2e8302a064622c484d612c3c48f8cedfc66fe3201f1ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/188b2721cbc4bb7917d2e8302a064622c484d612c3c48f8cedfc66fe3201f1ec/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b6a087600cbdcb9ce02b7a4e146e0dd49a26d361b2bfd58000df97cd080550b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b6a087600cbdcb9ce02b7a4e146e0dd49a26d361b2bfd58000df97cd080550b/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b7d7a6db961bfef862e90db2aca93842cf4863e6ab26c33af1b217ff4276fba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b7d7a6db961bfef862e90db2aca93842cf4863e6ab26c33af1b217ff4276fba/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/259bea6e5a944368b40bbd724270b44065932a6167fa1e8ccac33ec171b40c55/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/259bea6e5a944368b40bbd724270b44065932a6167fa1e8ccac33ec171b40c55/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26f431d8cc18618290f28183fa37a140bb0f72bcd41b7cb644247ff93ae84d30/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26f431d8cc18618290f28183fa37a140bb0f72bcd41b7cb644247ff93ae84d30/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/29f62b2ea7423ce44a08ae94c6c695b2e93bfbdb81d23930cffaab45c8db081c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/29f62b2ea7423ce44a08ae94c6c695b2e93bfbdb81d23930cffaab45c8db081c/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31582fbc125b37e50cc345dbd42e7047341b76afd096dd265541a5a55e9b5b94/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31582fbc125b37e50cc345dbd42e7047341b76afd096dd265541a5a55e9b5b94/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46a570e581b4fb9a1bc8a245abf0cc7cd235810158021bc29081243a3aa83256/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46a570e581b4fb9a1bc8a245abf0cc7cd235810158021bc29081243a3aa83256/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4dbbd0f8ac94c46de1949f712095f105b2dc8de0840e7561ee87316e9aec8483/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4dbbd0f8ac94c46de1949f712095f105b2dc8de0840e7561ee87316e9aec8483/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e14abf65dd39ec088575e0d8b9e2d5ca91f63b57b2f9080b3f7af18d1b1cdf0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e14abf65dd39ec088575e0d8b9e2d5ca91f63b57b2f9080b3f7af18d1b1cdf0/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6432ecd21ecbbc981dc032e54f0a8426520ef280116b9d595649f14ac25a7f07/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6432ecd21ecbbc981dc032e54f0a8426520ef280116b9d595649f14ac25a7f07/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6be2381599337d6580cefd43f997ec6eff8e7b9f84eefb7e4b1fa83a5c2be5e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6be2381599337d6580cefd43f997ec6eff8e7b9f84eefb7e4b1fa83a5c2be5e9/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a83146c0d573217f00980b7fd41c8c1b474aa36a050d75b280b82c8ece5681b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a83146c0d573217f00980b7fd41c8c1b474aa36a050d75b280b82c8ece5681b/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b24e062c3af897f8f9d7ea821a287050e2df1c7c14c1e7818c2ca7814d8edac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b24e062c3af897f8f9d7ea821a287050e2df1c7c14c1e7818c2ca7814d8edac/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/95ecdf2bd50b82bfda86de3b56e6c5a5f58481b3f4f64f69fd1abc173fbda723/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/95ecdf2bd50b82bfda86de3b56e6c5a5f58481b3f4f64f69fd1abc173fbda723/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc8c68976e0a46c1f637ff27ab50995b2c2befc957d10003c1bebef8c4c9a8a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc8c68976e0a46c1f637ff27ab50995b2c2befc957d10003c1bebef8c4c9a8a6/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dfd9aba9499701d8616e24cc1e3b84b531f788c45a805468f92b6e2f9440ce3f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dfd9aba9499701d8616e24cc1e3b84b531f788c45a805468f92b6e2f9440ce3f/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e2f05788772608ca6575554c6aa26b79b2e8da04675dbbcc33d6bd2e67eb5292/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e2f05788772608ca6575554c6aa26b79b2e8da04675dbbcc33d6bd2e67eb5292/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e8169e439f336e2ac9872ddad9e96cf2394c0d583a96b876ebdbc2b2efa91e32/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e8169e439f336e2ac9872ddad9e96cf2394c0d583a96b876ebdbc2b2efa91e32/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f00d7456ca3983460851aa61fdc8cfafa316a32c96c906e9f24ad72c5eedc0e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f00d7456ca3983460851aa61fdc8cfafa316a32c96c906e9f24ad72c5eedc0e9/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f682692a9619f3ebdb75e9843c1c09182742e713349b7cfb4b6d083bf870f951/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f682692a9619f3ebdb75e9843c1c09182742e713349b7cfb4b6d083bf870f951/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f99ebf26277a5267b307753c364de36f94b220110643560650df779472db037f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f99ebf26277a5267b307753c364de36f94b220110643560650df779472db037f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0388d722-6c0b-4cf0-9f0f-7ee369004008/volumes/kubernetes.io~projected/kube-api-access-m29ck:{mountpoint:/var/lib/kubelet/pods/0388d722-6c0b-4cf0-9f0f-7ee369004008/volumes/kubernetes.io~projected/kube-api-access-m29ck major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07a537c1-669a-4a02-bc1f-f09ea24e47ca/volumes/kubernetes.io~projected/kube-api-access-qxjln:{mountpoint:/var/lib/kubelet/pods/07a537c1-669a-4a02-bc1f-f09ea24e47ca/volumes/kubernetes.io~projected/kube-api-access-qxjln major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07a537c1-669a-4a02-bc1f-f09ea24e47ca/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/07a537c1-669a-4a02-bc1f-f09ea24e47ca/volumes/kubernetes.io~secret/cert major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0816e7fb-46c8-49d0-9c07-36fc49188b84/volumes/kubernetes.io~projected/kube-api-access-lv58t:{mountpoint:/var/lib/kubelet/pods/0816e7fb-46c8-49d0-9c07-36fc49188b84/volumes/kubernetes.io~projected/kube-api-access-lv58t major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e7f3fff-da9c-4189-bd53-5d6586ee0a4c/volumes/kubernetes.io~projected/kube-api-access-dm2gz:{mountpoint:/var/lib/kubelet/pods/0e7f3fff-da9c-4189-bd53-5d6586ee0a4c/volumes/kubernetes.io~projected/kube-api-access-dm2gz major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e7f3fff-da9c-4189-bd53-5d6586ee0a4c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/0e7f3fff-da9c-4189-bd53-5d6586ee0a4c/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0febea83-955d-4a56-8bc4-c883a4141109/volumes/kubernetes.io~projected/kube-api-access-ngw2c:{mountpoint:/var/lib/kubelet/pods/0febea83-955d-4a56-8bc4-c883a4141109/volumes/kubernetes.io~projected/kube-api-access-ngw2c major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1065c119-4756-47ad-960c-c99ee1fae963/volumes/kubernetes.io~projected/kube-api-access-whtg5:{mountpoint:/var/lib/kubelet/pods/1065c119-4756-47ad-960c-c99ee1fae963/volumes/kubernetes.io~projected/kube-api-access-whtg5 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1953d5bb-b35f-4c91-a263-a3664cf7fab2/volumes/kubernetes.io~projected/kube-api-access-2kr4j:{mountpoint:/var/lib/kubelet/pods/1953d5bb-b35f-4c91-a263-a3664cf7fab2/volumes/kubernetes.io~projected/kube-api-access-2kr4j major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1eb2f410-512d-4e3e-838a-87c1e3b47207/volumes/kubernetes.io~projected/kube-api-access-jtz7z:{mountpoint:/var/lib/kubelet/pods/1eb2f410-512d-4e3e-838a-87c1e3b47207/volumes/kubernetes.io~projected/kube-api-access-jtz7z major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/259e6b95-d64c-4101-871f-4a933d4ac38f/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/259e6b95-d64c-4101-871f-4a933d4ac38f/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/259e6b95-d64c-4101-871f-4a933d4ac38f/volumes/kubernetes.io~projected/kube-api-access-v5mps:{mountpoint:/var/lib/kubelet/pods/259e6b95-d64c-4101-871f-4a933d4ac38f/volumes/kubernetes.io~projected/kube-api-access-v5mps major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/259e6b95-d64c-4101-871f-4a933d4ac38f/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/259e6b95-d64c-4101-871f-4a933d4ac38f/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30af126f-08a0-49e2-9988-50b2f8380cfe/volumes/kubernetes.io~projected/kube-api-access-thktc:{mountpoint:/var/lib/kubelet/pods/30af126f-08a0-49e2-9988-50b2f8380cfe/volumes/kubernetes.io~projected/kube-api-access-thktc major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30af126f-08a0-49e2-9988-50b2f8380cfe/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/30af126f-08a0-49e2-9988-50b2f8380cfe/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b5fdb27-5000-4cf8-a025-13ae5389d17f/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/4b5fdb27-5000-4cf8-a025-13ae5389d17f/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b5fdb27-5000-4cf8-a025-13ae5389d17f/volumes/kubernetes.io~projected/kube-api-access-69xwz:{mountpoint:/var/lib/kubelet/pods/4b5fdb27-5000-4cf8-a025-13ae5389d17f/volumes/kubernetes.io~projected/kube-api-access-69xwz major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b5fdb27-5000-4cf8-a025-13ae5389d17f/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/4b5fdb27-5000-4cf8-a025-13ae5389d17f/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b5fdb27-5000-4cf8-a025-13ae5389d17f/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/4b5fdb27-5000-4cf8-a025-13ae5389d17f/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/530bea30-e132-4b1a-9c6d-e421c216060a/volumes/kubernetes.io~projected/kube-api-access-d2ml4:{mountpoint:/var/lib/kubelet/pods/530bea30-e132-4b1a-9c6d-e421c216060a/volumes/kubernetes.io~projected/kube-api-access-d2ml4 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97/volumes/kubernetes.io~projected/kube-api-access-8hs6g:{mountpoint:/var/lib/kubelet/pods/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97/volumes/kubernetes.io~projected/kube-api-access-8hs6g major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c0cf3cb-ed1a-453c-a3a4-9289b557ae3d/volumes/kubernetes.io~projected/kube-api-access-52fgx:{mountpoint:/var/lib/kubelet/pods/5c0cf3cb-ed1a-453c-a3a4-9289b557ae3d/volumes/kubernetes.io~projected/kube-api-access-52fgx major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/676904dc-4d4f-4bd6-b380-357d5d9a29d4/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/676904dc-4d4f-4bd6-b380-357d5d9a29d4/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73358782-c31a-4cb4-a0c2-377a1bceae4e/volumes/kubernetes.io~projected/kube-api-access-sxt72:{mountpoint:/var/lib/kubelet/pods/73358782-c31a-4cb4-a0c2-377a1bceae4e/volumes/kubernetes.io~projected/kube-api-access-sxt72 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81d7056d-8cf4-4058-a8fa-1731a0b93ff6/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/81d7056d-8cf4-4058-a8fa-1731a0b93ff6/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81d7056d-8cf4-4058-a8fa-1731a0b93ff6/volumes/kubernetes.io~projected/kube-api-access-xntk2:{mountpoint:/var/lib/kubelet/pods/81d7056d-8cf4-4058-a8fa-1731a0b93ff6/volumes/kubernetes.io~projected/kube-api-access-xntk2 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81d7056d-8cf4-4058-a8fa-1731a0b93ff6/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/81d7056d-8cf4-4058-a8fa-1731a0b93ff6/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81d7056d-8cf4-4058-a8fa-1731a0b93ff6/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/81d7056d-8cf4-4058-a8fa-1731a0b93ff6/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8428734b-4a9f-4df9-ac06-81074198a9ba/volumes/kubernetes.io~projected/kube-api-access-vfnx5:{mountpoint:/var/lib/kubelet/pods/8428734b-4a9f-4df9-ac06-81074198a9ba/volumes/kubernetes.io~projected/kube-api-access-vfnx5 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d45cab6-698a-4a61-be2c-58fe7d41c55c/volumes/kubernetes.io~projected/kube-api-access-mppj9:{mountpoint:/var/lib/kubelet/pods/8d45cab6-698a-4a61-be2c-58fe7d41c55c/volumes/kubernetes.io~projected/kube-api-access-mppj9 major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9400c4a1-ccf5-4b45-94f8-e37079762e4f/volumes/kubernetes.io~projected/kube-api-access-pbk9p:{mountpoint:/var/lib/kubelet/pods/9400c4a1-ccf5-4b45-94f8-e37079762e4f/volumes/kubernetes.io~projected/kube-api-access-pbk9p major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99c9881d-05b3-4953-b484-3ce51520b84a/volumes/kubernetes.io~projected/kube-api-access-z46kb:{mountpoint:/var/lib/kubelet/pods/99c9881d-05b3-4953-b484-3ce51520b84a/volumes/kubernetes.io~projected/kube-api-access-z46kb major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814/volumes/kubernetes.io~projected/kube-api-access-x75hw:{mountpoint:/var/lib/kubelet/pods/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814/volumes/kubernetes.io~projected/kube-api-access-x75hw major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2536550-3bec-4756-b872-7b14a8eb78c3/volumes/kubernetes.io~projected/kube-api-access-xqh8q:{mountpoint:/var/lib/kubelet/pods/a2536550-3bec-4756-b872-7b14a8eb78c3/volumes/kubernetes.io~projected/kube-api-access-xqh8q major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2536550-3bec-4756-b872-7b14a8eb78c3/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/a2536550-3bec-4756-b872-7b14a8eb78c3/volumes/kubernetes.io~secret/default-certificate major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2536550-3bec-4756-b872-7b14a8eb78c3/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/a2536550-3bec-4756-b872-7b14a8eb78c3/volumes/kubernetes.io~secret/stats-auth major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af572bfb-940b-4226-b91f-d06554b40d3c/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/af572bfb-940b-4226-b91f-d06554b40d3c/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b546ae66-9568-4840-a4e5-659d93eb80c5/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b546ae66-9568-4840-a4e5-659d93eb80c5/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b546ae66-9568-4840-a4e5-659d93eb80c5/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b546ae66-9568-4840-a4e5-659d93eb80c5/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b546ae66-9568-4840-a4e5-659d93eb80c5/volumes/kubernetes.io~projected/kube-api-access-x7dvt:{mountpoint:/var/lib/kubelet/pods/b546ae66-9568-4840-a4e5-659d93eb80c5/volumes/kubernetes.io~projected/kube-api-access-x7dvt major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce551e36-0249-4723-b814-a98f0a282eed/volumes/kubernetes.io~projected/kube-api-access-62lfn:{mountpoint:/var/lib/kubelet/pods/ce551e36-0249-4723-b814-a98f0a282eed/volumes/kubernetes.io~projected/kube-api-access-62lfn major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d172a43b-a20c-4fc3-a1cc-77a697312061/volumes/kubernetes.io~projected/kube-api-access-8kmlt:{mountpoint:/var/lib/kubelet/pods/d172a43b-a20c-4fc3-a1cc-77a697312061/volumes/kubernetes.io~projected/kube-api-access-8kmlt major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/volumes/kubernetes.io~projected/kube-api-access-v9z57:{mountpoint:/var/lib/kubelet/pods/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/volumes/kubernetes.io~projected/kube-api-access-v9z57 major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f934ff4f-dadc-469b-a87e-16db95e3181f/volumes/kubernetes.io~projected/kube-api-access-brg9z:{mountpoint:/var/lib/kubelet/pods/f934ff4f-dadc-469b-a87e-16db95e3181f/volumes/kubernetes.io~projected/kube-api-access-brg9z major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc992647-89dd-44cc-ba19-3fdb03e125de/volumes/kubernetes.io~projected/kube-api-access-9bhr9:{mountpoint:/var/lib/kubelet/pods/fc992647-89dd-44cc-ba19-3fdb03e125de/volumes/kubernetes.io~projected/kube-api-access-9bhr9 major:0 minor:64 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/af508a3fae3ab113a0987088778be801180121a764b2dd931a75ee3c6f0534f6/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/86a065a59ef19213ecadf34dc507ab4c96aacb184d4359b789be7ecefcf8444e/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/e28f26f2f7a9666ddcccfcd6a20c535b5e2704f0f65553a65cc0a335ef41d93b/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/80d6aab322ff08c18126d11bf12a6dd0d90afeefed9fe5b147eacdb3a5d9b910/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b78cf50a045474f9f9eed122e8ca369dc6109c179cfaf3a87d287d1213fafcb2/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/edb346590cc8026e96af5bde2de5b4640fa3bac8edf16892e968eaec5a855e71/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/24a915e0db9713e1890cb5042b742c40bdbdd6ba9b015165f674d6360928a6dc/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/9005ad552d12523569d93a7e2b2d18eb95ec68c09bdb772ae578d753f005b4b3/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/ed8b7996ae742252f04255bc63d163f349678ca9f1b118bfa4bf4ba22e6a3d96/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/4378d6b1649008b8d5f14671683c4833b542f6bc29b0eb4aaa1a671d34ca3005/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/374e905b3742c0b92f43c0ac8785edc00dab01a0ca8c4c097edd1f5e1059c79f/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/4e3e242cc7092bb39f4d199d24bcf1aff2bf4a5ab2511cc73e207ad08d0378d6/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/798f7dac2f0ac5ffbc016f20b9c5b3d3493c0fb4853b650bd3182fd587654889/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/e5947b67b0bf8b91df1c93a0440fede4eca0b7c97e7e3dfc93ed373ce09bdf8a/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/6744bba82cb9845eaf0669f48bf27486d4dcc481cffa593a4a69546498a431ce/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/385e2ff41b7c098e44d27936cefd7c0d458d8404bf498856c4bf5154a6475800/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/ec3e6e35dd9afc30da25f57c9fdd62f5f1d5e09606b790aed579b5338eacfd39/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-178:{mountpoint:/var/lib/containers/storage/overlay/982215018c767e059ef19ecd0d710b21ef847b27ea53876ac84bfc43d1f9e925/merged major:0 minor:178 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/8441dada448964db85a9cbdb91f30c44480f618a8971c7325864375e5882e075/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e8011daabee4679a27a4958408bfb0d344eaa10b61c3f8579637123ac716fb9d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/bce262676af6034decf76937f8e91ddce129bbb04069e9bc2923df680dc15414/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/25d9ca6de31fa027dd2a77984bfadfa747be56679ea92b54efed46d79dad0976/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/26e88b752c085513df9e05fe037d5528023eb3ae0abbd956538dcdee4f33e2fc/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/071c05e7e2a79758c1331a3c2a9bb35c61ee51d181df33ebbaeac184bb86ff2e/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/b4673b4b0593e180a76b6781cd802544a5e973a4c202107725db906c2660a5cd/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/fe0ca5377bc542cc45395d15606c45b4de63f26bc862cd0ef798e5d6924b5281/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/d980e5dec4dba5b9b7e72740d362c67ae57abc56fd0b40e6166c96d0c571f516/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/94e29d9985979f5efb686f25837031b1e731afab4463826a7553724fe75abf35/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/b5408d6961f9683d888dbf1636eebf74c49b86efcd0c50a72bffd6019c102d17/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/8ef86c246a9646241acff208ff53d57b86fbad84ea37a652eaa86cdec156a5a9/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/50025b5a356c804a3415800eced17e1abf0964589869e74f135660c38da2027b/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/1a4bf67952a43289d736984464d12bdf8436afc7f130895797121a26ae1fadcf/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/366eace5e3811129d9edb9605a591eaee78400a86d1628ef2c7d7096f3b4a5d6/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/fc23311d92310205bb397a5e30644b56e9254b2b86ae6accc42bfd985ca57031/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/a7fed5c39a0f9e49dca60cd1fe9c23cbf55008baa2dac705ae1893afe63e633e/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/e302bf7d24238a5c6a54046ca5cd3803e4a5eb8d3f8bd42031ba3ad6a4fe0bf6/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/ec73223a4695486fd264f7127893f813e1b81c3b6ef6bb8bc45735a9ee220ba0/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/43483976c8f31716dcaabb31723656380809b08feb5dd41dae30d1a0d0b9edce/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/82bb705688de00658b00790bf1cee0e91461e0872c0e839e1858261412206e4c/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/a9495a8d373a6f6fe1596d37f2f36aa57da914f9c34a10839234cb58c5dd7591/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/76adb39e6da84ab5d46077962b1c89e1a7bc4029a8794a83668677859066f696/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/9f6f6580c67c374b47af4dafe2280aa49687e959d699a0d47fd0bff7c4987a18/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/738265663be43de3aab1a028127cf09f6262d48c26043d83300e9141adc7f673/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/184166de41742bf5739390c85c44deb3bf6bf587ec30811050c344fa5faf2941/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/f494a1c4b50f31aa05851dfa061f1d539ab3fe7e297211ae41cc0237c4528f44/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/ef363f11e92c8c3bef041675157dab11be1cd29dc5f34345bb048ca701590f9d/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/48364f7580174619d23fa0be53b7a34fa69d910be89c071b2e98ee1be3fa2dad/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/cc676220f881c5bcedaeba33abcdcb1f49d4708750366d43f8fc2da27375e8d6/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/fc26860cf90549bf12a07dbffd6d6a0e47e5b7297831e79559b3866412e7648f/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/db4b1ceb01a314b60a8bb438ce1995e490790da35c63e4ee47506b89220340ac/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/d1215e6678870c02a38fcd663d5264f83c3df4ef00c2cf37231df8f9d9fdc3c7/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/1d847099ca6afb807a3f86a856a5ddc513582e2bdaa77026e1df91cae899823b/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/b01cf76440b1e4a9eed155d8a79448a5f8761c83f99588df8286578a4b764709/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/a932807b0ca775e17d0037543d7e0c28d0e69bccd075be6d92eca6c2646d30bc/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/7e9b3a5ea9aa8a551460ac3fa39531165e2cb0640ded0bc3f9ded83290ed8503/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/44e89f74666a65afdc72a9743e477b71b21ba91ececfa3240e1bc03c1e72095b/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 29 21:46:18.991478 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.986929    4962 manager.go:217] Machine: {Timestamp:2026-10-29 21:46:18.986140865 +0000 UTC m=+0.059097822 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3257184 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22465b80dda8f7b658a3ae5cef31e2 SystemUUID:ec22465b-80dd-a8f7-b658-a3ae5cef31e2 BootID:fbea307b-1efb-4aea-8948-138de3188448 Filesystems:[{Device:/run/containers/storage/overlay-containers/6432ecd21ecbbc981dc032e54f0a8426520ef280116b9d595649f14ac25a7f07/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/81d7056d-8cf4-4058-a8fa-1731a0b93ff6/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/056b9baf976e19e2257192d2af06ff59d4fab0e77ec5e24badf9543b14134834/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f934ff4f-dadc-469b-a87e-16db95e3181f/volumes/kubernetes.io~projected/kube-api-access-brg9z DeviceMajor:0 DeviceMinor:295 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/259e6b95-d64c-4101-871f-4a933d4ac38f/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1953d5bb-b35f-4c91-a263-a3664cf7fab2/volumes/kubernetes.io~projected/kube-api-access-2kr4j DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f99ebf26277a5267b307753c364de36f94b220110643560650df779472db037f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-178 DeviceMajor:0 DeviceMinor:178 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4b5fdb27-5000-4cf8-a025-13ae5389d17f/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8428734b-4a9f-4df9-ac06-81074198a9ba/volumes/kubernetes.io~projected/kube-api-access-vfnx5 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/30af126f-08a0-49e2-9988-50b2f8380cfe/volumes/kubernetes.io~projected/kube-api-access-thktc DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4b5fdb27-5000-4cf8-a025-13ae5389d17f/volumes/kubernetes.io~projected/kube-api-access-69xwz DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26f431d8cc18618290f28183fa37a140bb0f72bcd41b7cb644247ff93ae84d30/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af572bfb-940b-4226-b91f-d06554b40d3c/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0816e7fb-46c8-49d0-9c07-36fc49188b84/volumes/kubernetes.io~projected/kube-api-access-lv58t DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31582fbc125b37e50cc345dbd42e7047341b76afd096dd265541a5a55e9b5b94/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b7d7a6db961bfef862e90db2aca93842cf4863e6ab26c33af1b217ff4276fba/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dfd9aba9499701d8616e24cc1e3b84b531f788c45a805468f92b6e2f9440ce3f/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/29f62b2ea7423ce44a08ae94c6c695b2e93bfbdb81d23930cffaab45c8db081c/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/259e6b95-d64c-4101-871f-4a933d4ac38f/volumes/kubernetes.io~projected/kube-api-access-v5mps DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0febea83-955d-4a56-8bc4-c883a4141109/volumes/kubernetes.io~projected/kube-api-access-ngw2c DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/81d7056d-8cf4-4058-a8fa-1731a0b93ff6/volumes/kubernetes.io~projected/kube-api-access-xntk2 DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1065c119-4756-47ad-960c-c99ee1fae963/volumes/kubernetes.io~projected/kube-api-access-whtg5 DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a2536550-3bec-4756-b872-7b14a8eb78c3/volumes/kubernetes.io~projected/kube-api-access-xqh8q DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d45cab6-698a-4a61-be2c-58fe7d41c55c/volumes/kubernetes.io~projected/kube-api-access-mppj9 DeviceMajor:0 DeviceMinor:337 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b546ae66-9568-4840-a4e5-659d93eb80c5/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9400c4a1-ccf5-4b45-94f8-e37079762e4f/volumes/kubernetes.io~projected/kube-api-access-pbk9p DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81d7056d-8cf4-4058-a8fa-1731a0b93ff6/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5c0cf3cb-ed1a-453c-a3a4-9289b557ae3d/volumes/kubernetes.io~projected/kube-api-access-52fgx DeviceMajor:0 DeviceMinor:318 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e7f3fff-da9c-4189-bd53-5d6586ee0a4c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0388d722-6c0b-4cf0-9f0f-7ee369004008/volumes/kubernetes.io~projected/kube-api-access-m29ck DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e14abf65dd39ec088575e0d8b9e2d5ca91f63b57b2f9080b3f7af18d1b1cdf0/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6be2381599337d6580cefd43f997ec6eff8e7b9f84eefb7e4b1fa83a5c2be5e9/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/73358782-c31a-4cb4-a0c2-377a1bceae4e/volumes/kubernetes.io~projected/kube-api-access-sxt72 DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/188b2721cbc4bb7917d2e8302a064622c484d612c3c48f8cedfc66fe3201f1ec/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/676904dc-4d4f-4bd6-b380-357d5d9a29d4/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/30af126f-08a0-49e2-9988-50b2f8380cfe/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a83146c0d573217f00980b7fd41c8c1b474aa36a050d75b280b82c8ece5681b/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46a570e581b4fb9a1bc8a245abf0cc7cd235810158021bc29081243a3aa83256/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4dbbd0f8ac94c46de1949f712095f105b2dc8de0840e7561ee87316e9aec8483/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b24e062c3af897f8f9d7ea821a287050e2df1c7c14c1e7818c2ca7814d8edac/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/259e6b95-d64c-4101-871f-4a933d4ac38f/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e2f05788772608ca6575554c6aa26b79b2e8da04675dbbcc33d6bd2e67eb5292/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/259bea6e5a944368b40bbd724270b44065932a6167fa1e8ccac33ec171b40c55/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc8c68976e0a46c1f637ff27ab50995b2c2befc957d10003c1bebef8c4c9a8a6/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d172a43b-a20c-4fc3-a1cc-77a697312061/volumes/kubernetes.io~projected/kube-api-access-8kmlt DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e8169e439f336e2ac9872ddad9e96cf2394c0d583a96b876ebdbc2b2efa91e32/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814/volumes/kubernetes.io~projected/kube-api-access-x75hw DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1eb2f410-512d-4e3e-838a-87c1e3b47207/volumes/kubernetes.io~projected/kube-api-access-jtz7z DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f682692a9619f3ebdb75e9843c1c09182742e713349b7cfb4b6d083bf870f951/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2536550-3bec-4756-b872-7b14a8eb78c3/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/volumes/kubernetes.io~projected/kube-api-access-v9z57 DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4b5fdb27-5000-4cf8-a025-13ae5389d17f/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/95ecdf2bd50b82bfda86de3b56e6c5a5f58481b3f4f64f69fd1abc173fbda723/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0e7f3fff-da9c-4189-bd53-5d6586ee0a4c/volumes/kubernetes.io~projected/kube-api-access-dm2gz DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b6a087600cbdcb9ce02b7a4e146e0dd49a26d361b2bfd58000df97cd080550b/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2536550-3bec-4756-b872-7b14a8eb78c3/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/07a537c1-669a-4a02-bc1f-f09ea24e47ca/volumes/kubernetes.io~projected/kube-api-access-qxjln DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f00d7456ca3983460851aa61fdc8cfafa316a32c96c906e9f24ad72c5eedc0e9/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/07a537c1-669a-4a02-bc1f-f09ea24e47ca/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81d7056d-8cf4-4058-a8fa-1731a0b93ff6/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc992647-89dd-44cc-ba19-3fdb03e125de/volumes/kubernetes.io~projected/kube-api-access-9bhr9 DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b546ae66-9568-4840-a4e5-659d93eb80c5/volumes/kubernetes.io~projected/kube-api-access-x7dvt DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97/volumes/kubernetes.io~projected/kube-api-access-8hs6g DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/99c9881d-05b3-4953-b484-3ce51520b84a/volumes/kubernetes.io~projected/kube-api-access-z46kb DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/530bea30-e132-4b1a-9c6d-e421c216060a/volumes/kubernetes.io~projected/kube-api-access-d2ml4 DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b546ae66-9568-4840-a4e5-659d93eb80c5/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ce551e36-0249-4723-b814-a98f0a282eed/volumes/kubernetes.io~projected/kube-api-access-62lfn DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4b5fdb27-5000-4cf8-a025-13ae5389d17f/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:056b9baf976e19e MacAddress:d2:86:52:90:5e:0a Speed:10000 Mtu:8901} {Name:188b2721cbc4bb7 MacAddress:32:e8:7a:48:7e:6b Speed:10000 Mtu:8901} {Name:1b6a087600cbdcb MacAddress:d6:c2:df:7c:fd:c3 Speed:10000 Mtu:8901} {Name:259bea6e5a94436 MacAddress:4e:82:34:3c:a8:76 Speed:10000 Mtu:8901} {Name:26f431d8cc18618 MacAddress:86:08:a9:e7:9c:0d Speed:10000 Mtu:8901} {Name:31582fbc125b37e MacAddress:46:d8:22:7f:4d:de Speed:10000 Mtu:8901} {Name:46a570e581b4fb9 MacAddress:c2:d9:05:4b:cc:30 Speed:10000 Mtu:8901} {Name:4dbbd0f8ac94c46 MacAddress:82:15:1c:0a:c4:92 Speed:10000 Mtu:8901} {Name:4e14abf65dd39ec MacAddress:82:f0:20:86:9f:ec Speed:10000 Mtu:8901} {Name:8b24e062c3af897 MacAddress:ce:70:c4:cd:99:d5 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c2:55:79:e1 Speed:0 Mtu:9001} {Name:br-int MacAddress:2e:7a:2e:0c:b9:e8 Speed:0 Mtu:8901} {Name:e8169e439f336e2 MacAddress:de:f0:b5:a3:da:66 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c2:55:79:e1 Speed:0 Mtu:9001} {Name:f00d7456ca39834 MacAddress:7a:f1:89:71:67:59 Speed:10000 Mtu:8901} {Name:f682692a9619f3e MacAddress:4e:ca:2b:7c:f3:9b Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:72:ba:ab:5c:cb:6d Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:7a:ac:61:73:62:e3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 29 21:46:18.991478 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.987552    4962 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 29 21:46:18.991478 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.987608    4962 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 29 21:46:18.991478 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.987881    4962 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 21:46:18.991478 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.987906    4962 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-146.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 29 21:46:18.991478 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.988019    4962 topology_manager.go:143] "Creating topology manager with none policy"
Oct 29 21:46:18.991478 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.988026    4962 container_manager_linux.go:311] "Creating device plugin manager"
Oct 29 21:46:18.991478 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.988035    4962 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 21:46:18.991478 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.988048    4962 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 21:46:18.991478 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.988064    4962 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 29 21:46:18.991478 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.988207    4962 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 29 21:46:18.991478 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.988318    4962 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 29 21:46:18.991478 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.988350    4962 kubelet.go:493] "Attempting to sync node with API server"
Oct 29 21:46:18.991478 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.988358    4962 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 29 21:46:18.991478 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.988368    4962 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 29 21:46:18.991478 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.988376    4962 kubelet.go:405] "Adding apiserver pod source"
Oct 29 21:46:18.991478 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.988384    4962 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 21:46:18.991478 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.989224    4962 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 29 21:46:18.991478 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.989234    4962 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 21:46:18.991478 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.989697    4962 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 29 21:46:18.992639 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.991727    4962 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 29 21:46:18.992639 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.991736    4962 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 21:46:18.992639 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.991835    4962 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 29 21:46:18.992639 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.992013    4962 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 29 21:46:18.992639 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.992039    4962 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 29 21:46:18.992639 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.992141    4962 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 29 21:46:18.992639 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.992152    4962 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 29 21:46:18.992639 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.992159    4962 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 29 21:46:18.992639 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.992164    4962 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 29 21:46:18.992639 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.992173    4962 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 29 21:46:18.992639 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.992182    4962 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 29 21:46:18.992639 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.992189    4962 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 29 21:46:18.992639 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.992195    4962 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 29 21:46:18.992639 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.992202    4962 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 29 21:46:18.992639 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.992208    4962 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 29 21:46:18.992639 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.992216    4962 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 29 21:46:18.992639 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.992225    4962 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 29 21:46:18.992639 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.992248    4962 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 29 21:46:18.992639 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.992256    4962 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 29 21:46:18.995483 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.995468    4962 server.go:1263] "Started kubelet"
Oct 29 21:46:19.007332 ip-10-0-129-146 systemd[1]: Started Kubernetes Kubelet.
Oct 29 21:46:19.010186 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:18.995628    4962 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 29 21:46:19.010186 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.007752    4962 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 29 21:46:19.010186 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.008087    4962 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 29 21:46:19.010186 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.007396    4962 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 21:46:19.011728 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.011710    4962 server.go:317] "Adding debug handlers to kubelet server"
Oct 29 21:46:19.012021 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.011999    4962 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 29 21:46:19.012454 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.012433    4962 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 29 21:46:19.012630 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.012606    4962 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 29 21:46:19.012630 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.012625    4962 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 29 21:46:19.012766 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.012730    4962 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 29 21:46:19.013610 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:19.013587    4962 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-146.ec2.internal\" not found"
Oct 29 21:46:19.016154 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.014311    4962 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 21:40:40 +0000 UTC" deadline="2028-05-26 19:44:41.847122946 +0000 UTC"
Oct 29 21:46:19.016154 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.014351    4962 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13797h58m22.832775502s"
Oct 29 21:46:19.016154 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.015627    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81d7056d-8cf4-4058-a8fa-1731a0b93ff6" volumeName="kubernetes.io/secret/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-installation-pull-secrets" seLinuxMountContext=""
Oct 29 21:46:19.016154 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.015661    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2536550-3bec-4756-b872-7b14a8eb78c3" volumeName="kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-stats-auth" seLinuxMountContext=""
Oct 29 21:46:19.016154 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.015676    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600" volumeName="kubernetes.io/secret/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600-serving-cert" seLinuxMountContext=""
Oct 29 21:46:19.016154 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.015689    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e7f3fff-da9c-4189-bd53-5d6586ee0a4c" volumeName="kubernetes.io/secret/0e7f3fff-da9c-4189-bd53-5d6586ee0a4c-serving-cert" seLinuxMountContext=""
Oct 29 21:46:19.016154 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.015703    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="259e6b95-d64c-4101-871f-4a933d4ac38f" volumeName="kubernetes.io/configmap/259e6b95-d64c-4101-871f-4a933d4ac38f-ovnkube-config" seLinuxMountContext=""
Oct 29 21:46:19.016870 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.016836    4962 factory.go:55] Registering systemd factory
Oct 29 21:46:19.016870 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.016856    4962 factory.go:223] Registration of the systemd container factory successfully
Oct 29 21:46:19.017555 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.017540    4962 factory.go:153] Registering CRI-O factory
Oct 29 21:46:19.017643 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.017557    4962 factory.go:223] Registration of the crio container factory successfully
Oct 29 21:46:19.017643 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.017606    4962 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 29 21:46:19.017643 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.017630    4962 factory.go:103] Registering Raw factory
Oct 29 21:46:19.017643 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.017645    4962 manager.go:1199] Started watching for new ooms in manager
Oct 29 21:46:19.017968 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.017956    4962 manager.go:319] Starting recovery of all containers
Oct 29 21:46:19.019594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019569    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="259e6b95-d64c-4101-871f-4a933d4ac38f" volumeName="kubernetes.io/secret/259e6b95-d64c-4101-871f-4a933d4ac38f-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 29 21:46:19.019594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019599    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9400c4a1-ccf5-4b45-94f8-e37079762e4f" volumeName="kubernetes.io/configmap/9400c4a1-ccf5-4b45-94f8-e37079762e4f-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 29 21:46:19.019752 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019608    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9400c4a1-ccf5-4b45-94f8-e37079762e4f" volumeName="kubernetes.io/configmap/9400c4a1-ccf5-4b45-94f8-e37079762e4f-cni-binary-copy" seLinuxMountContext=""
Oct 29 21:46:19.019752 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019623    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99c9881d-05b3-4953-b484-3ce51520b84a" volumeName="kubernetes.io/projected/99c9881d-05b3-4953-b484-3ce51520b84a-kube-api-access-z46kb" seLinuxMountContext=""
Oct 29 21:46:19.019752 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019636    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600" volumeName="kubernetes.io/configmap/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600-config" seLinuxMountContext=""
Oct 29 21:46:19.019752 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019651    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07a537c1-669a-4a02-bc1f-f09ea24e47ca" volumeName="kubernetes.io/projected/07a537c1-669a-4a02-bc1f-f09ea24e47ca-kube-api-access-qxjln" seLinuxMountContext=""
Oct 29 21:46:19.019752 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019663    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0816e7fb-46c8-49d0-9c07-36fc49188b84" volumeName="kubernetes.io/projected/0816e7fb-46c8-49d0-9c07-36fc49188b84-kube-api-access-lv58t" seLinuxMountContext=""
Oct 29 21:46:19.019752 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019671    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1eb2f410-512d-4e3e-838a-87c1e3b47207" volumeName="kubernetes.io/configmap/1eb2f410-512d-4e3e-838a-87c1e3b47207-telemetry-config" seLinuxMountContext=""
Oct 29 21:46:19.019752 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019679    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="676904dc-4d4f-4bd6-b380-357d5d9a29d4" volumeName="kubernetes.io/secret/676904dc-4d4f-4bd6-b380-357d5d9a29d4-original-pull-secret" seLinuxMountContext=""
Oct 29 21:46:19.019752 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019689    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73358782-c31a-4cb4-a0c2-377a1bceae4e" volumeName="kubernetes.io/projected/73358782-c31a-4cb4-a0c2-377a1bceae4e-kube-api-access-sxt72" seLinuxMountContext=""
Oct 29 21:46:19.019752 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019698    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81d7056d-8cf4-4058-a8fa-1731a0b93ff6" volumeName="kubernetes.io/configmap/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-certificates" seLinuxMountContext=""
Oct 29 21:46:19.019752 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019715    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b5fdb27-5000-4cf8-a025-13ae5389d17f" volumeName="kubernetes.io/configmap/4b5fdb27-5000-4cf8-a025-13ae5389d17f-trusted-ca" seLinuxMountContext=""
Oct 29 21:46:19.019752 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019723    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81d7056d-8cf4-4058-a8fa-1731a0b93ff6" volumeName="kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-bound-sa-token" seLinuxMountContext=""
Oct 29 21:46:19.019752 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019732    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9400c4a1-ccf5-4b45-94f8-e37079762e4f" volumeName="kubernetes.io/configmap/9400c4a1-ccf5-4b45-94f8-e37079762e4f-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 29 21:46:19.019752 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019739    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2536550-3bec-4756-b872-7b14a8eb78c3" volumeName="kubernetes.io/projected/a2536550-3bec-4756-b872-7b14a8eb78c3-kube-api-access-xqh8q" seLinuxMountContext=""
Oct 29 21:46:19.019752 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019746    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b5fdb27-5000-4cf8-a025-13ae5389d17f" volumeName="kubernetes.io/configmap/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-certificates" seLinuxMountContext=""
Oct 29 21:46:19.019752 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019753    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97" volumeName="kubernetes.io/configmap/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-service-ca-bundle" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019762    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0febea83-955d-4a56-8bc4-c883a4141109" volumeName="kubernetes.io/projected/0febea83-955d-4a56-8bc4-c883a4141109-kube-api-access-ngw2c" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019798    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1eb2f410-512d-4e3e-838a-87c1e3b47207" volumeName="kubernetes.io/projected/1eb2f410-512d-4e3e-838a-87c1e3b47207-kube-api-access-jtz7z" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019811    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c0cf3cb-ed1a-453c-a3a4-9289b557ae3d" volumeName="kubernetes.io/projected/5c0cf3cb-ed1a-453c-a3a4-9289b557ae3d-kube-api-access-52fgx" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019823    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600" volumeName="kubernetes.io/projected/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600-kube-api-access-v9z57" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019835    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="259e6b95-d64c-4101-871f-4a933d4ac38f" volumeName="kubernetes.io/projected/259e6b95-d64c-4101-871f-4a933d4ac38f-kube-api-access-v5mps" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019847    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30af126f-08a0-49e2-9988-50b2f8380cfe" volumeName="kubernetes.io/secret/30af126f-08a0-49e2-9988-50b2f8380cfe-serving-cert" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019867    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81d7056d-8cf4-4058-a8fa-1731a0b93ff6" volumeName="kubernetes.io/configmap/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-trusted-ca" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019879    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99c9881d-05b3-4953-b484-3ce51520b84a" volumeName="kubernetes.io/empty-dir/99c9881d-05b3-4953-b484-3ce51520b84a-tmp-dir" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019892    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc992647-89dd-44cc-ba19-3fdb03e125de" volumeName="kubernetes.io/empty-dir/fc992647-89dd-44cc-ba19-3fdb03e125de-tmp-dir" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019906    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e7f3fff-da9c-4189-bd53-5d6586ee0a4c" volumeName="kubernetes.io/projected/0e7f3fff-da9c-4189-bd53-5d6586ee0a4c-kube-api-access-dm2gz" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019918    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="259e6b95-d64c-4101-871f-4a933d4ac38f" volumeName="kubernetes.io/configmap/259e6b95-d64c-4101-871f-4a933d4ac38f-env-overrides" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019930    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97" volumeName="kubernetes.io/empty-dir/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-tmp" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019944    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8428734b-4a9f-4df9-ac06-81074198a9ba" volumeName="kubernetes.io/configmap/8428734b-4a9f-4df9-ac06-81074198a9ba-serviceca" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019957    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814" volumeName="kubernetes.io/configmap/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-cni-binary-copy" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019968    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2536550-3bec-4756-b872-7b14a8eb78c3" volumeName="kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-default-certificate" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.019984    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b546ae66-9568-4840-a4e5-659d93eb80c5" volumeName="kubernetes.io/empty-dir/b546ae66-9568-4840-a4e5-659d93eb80c5-tmp" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020009    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99c9881d-05b3-4953-b484-3ce51520b84a" volumeName="kubernetes.io/configmap/99c9881d-05b3-4953-b484-3ce51520b84a-config-volume" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020021    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30af126f-08a0-49e2-9988-50b2f8380cfe" volumeName="kubernetes.io/configmap/30af126f-08a0-49e2-9988-50b2f8380cfe-config" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020035    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81d7056d-8cf4-4058-a8fa-1731a0b93ff6" volumeName="kubernetes.io/empty-dir/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-ca-trust-extracted" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020050    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81d7056d-8cf4-4058-a8fa-1731a0b93ff6" volumeName="kubernetes.io/secret/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-image-registry-private-configuration" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020065    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9400c4a1-ccf5-4b45-94f8-e37079762e4f" volumeName="kubernetes.io/projected/9400c4a1-ccf5-4b45-94f8-e37079762e4f-kube-api-access-pbk9p" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020077    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af572bfb-940b-4226-b91f-d06554b40d3c" volumeName="kubernetes.io/configmap/af572bfb-940b-4226-b91f-d06554b40d3c-konnectivity-ca" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020089    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b546ae66-9568-4840-a4e5-659d93eb80c5" volumeName="kubernetes.io/projected/b546ae66-9568-4840-a4e5-659d93eb80c5-kube-api-access-x7dvt" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020101    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97" volumeName="kubernetes.io/configmap/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-trusted-ca-bundle" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020115    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30af126f-08a0-49e2-9988-50b2f8380cfe" volumeName="kubernetes.io/projected/30af126f-08a0-49e2-9988-50b2f8380cfe-kube-api-access-thktc" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020128    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b5fdb27-5000-4cf8-a025-13ae5389d17f" volumeName="kubernetes.io/secret/4b5fdb27-5000-4cf8-a025-13ae5389d17f-installation-pull-secrets" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020141    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97" volumeName="kubernetes.io/projected/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-kube-api-access-8hs6g" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020153    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97" volumeName="kubernetes.io/secret/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-serving-cert" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020166    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600" volumeName="kubernetes.io/configmap/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600-trusted-ca" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020220    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f934ff4f-dadc-469b-a87e-16db95e3181f" volumeName="kubernetes.io/projected/f934ff4f-dadc-469b-a87e-16db95e3181f-kube-api-access-brg9z" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020241    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0388d722-6c0b-4cf0-9f0f-7ee369004008" volumeName="kubernetes.io/configmap/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-config" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020254    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b5fdb27-5000-4cf8-a025-13ae5389d17f" volumeName="kubernetes.io/secret/4b5fdb27-5000-4cf8-a025-13ae5389d17f-image-registry-private-configuration" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020269    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="530bea30-e132-4b1a-9c6d-e421c216060a" volumeName="kubernetes.io/projected/530bea30-e132-4b1a-9c6d-e421c216060a-kube-api-access-d2ml4" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020282    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d45cab6-698a-4a61-be2c-58fe7d41c55c" volumeName="kubernetes.io/empty-dir/8d45cab6-698a-4a61-be2c-58fe7d41c55c-data-volume" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020357    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814" volumeName="kubernetes.io/projected/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-kube-api-access-x75hw" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020373    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07a537c1-669a-4a02-bc1f-f09ea24e47ca" volumeName="kubernetes.io/secret/07a537c1-669a-4a02-bc1f-f09ea24e47ca-cert" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020393    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1953d5bb-b35f-4c91-a263-a3664cf7fab2" volumeName="kubernetes.io/configmap/1953d5bb-b35f-4c91-a263-a3664cf7fab2-iptables-alerter-script" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020406    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1953d5bb-b35f-4c91-a263-a3664cf7fab2" volumeName="kubernetes.io/projected/1953d5bb-b35f-4c91-a263-a3664cf7fab2-kube-api-access-2kr4j" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020419    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81d7056d-8cf4-4058-a8fa-1731a0b93ff6" volumeName="kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-kube-api-access-xntk2" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020433    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d45cab6-698a-4a61-be2c-58fe7d41c55c" volumeName="kubernetes.io/projected/8d45cab6-698a-4a61-be2c-58fe7d41c55c-kube-api-access-mppj9" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020448    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af572bfb-940b-4226-b91f-d06554b40d3c" volumeName="kubernetes.io/secret/af572bfb-940b-4226-b91f-d06554b40d3c-agent-certs" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020462    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce551e36-0249-4723-b814-a98f0a282eed" volumeName="kubernetes.io/projected/ce551e36-0249-4723-b814-a98f0a282eed-kube-api-access-62lfn" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020475    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d172a43b-a20c-4fc3-a1cc-77a697312061" volumeName="kubernetes.io/projected/d172a43b-a20c-4fc3-a1cc-77a697312061-kube-api-access-8kmlt" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020487    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b5fdb27-5000-4cf8-a025-13ae5389d17f" volumeName="kubernetes.io/empty-dir/4b5fdb27-5000-4cf8-a025-13ae5389d17f-ca-trust-extracted" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020502    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b5fdb27-5000-4cf8-a025-13ae5389d17f" volumeName="kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-bound-sa-token" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020516    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b5fdb27-5000-4cf8-a025-13ae5389d17f" volumeName="kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-kube-api-access-69xwz" seLinuxMountContext=""
Oct 29 21:46:19.020496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020529    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0388d722-6c0b-4cf0-9f0f-7ee369004008" volumeName="kubernetes.io/projected/0388d722-6c0b-4cf0-9f0f-7ee369004008-kube-api-access-m29ck" seLinuxMountContext=""
Oct 29 21:46:19.022723 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020538    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e7f3fff-da9c-4189-bd53-5d6586ee0a4c" volumeName="kubernetes.io/configmap/0e7f3fff-da9c-4189-bd53-5d6586ee0a4c-config" seLinuxMountContext=""
Oct 29 21:46:19.022723 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020546    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97" volumeName="kubernetes.io/empty-dir/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-snapshots" seLinuxMountContext=""
Oct 29 21:46:19.022723 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020556    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8428734b-4a9f-4df9-ac06-81074198a9ba" volumeName="kubernetes.io/projected/8428734b-4a9f-4df9-ac06-81074198a9ba-kube-api-access-vfnx5" seLinuxMountContext=""
Oct 29 21:46:19.022723 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020564    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1065c119-4756-47ad-960c-c99ee1fae963" volumeName="kubernetes.io/projected/1065c119-4756-47ad-960c-c99ee1fae963-kube-api-access-whtg5" seLinuxMountContext=""
Oct 29 21:46:19.022723 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020571    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814" volumeName="kubernetes.io/configmap/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-multus-daemon-config" seLinuxMountContext=""
Oct 29 21:46:19.022723 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020579    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b546ae66-9568-4840-a4e5-659d93eb80c5" volumeName="kubernetes.io/empty-dir/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-tuned" seLinuxMountContext=""
Oct 29 21:46:19.022723 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020586    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc992647-89dd-44cc-ba19-3fdb03e125de" volumeName="kubernetes.io/projected/fc992647-89dd-44cc-ba19-3fdb03e125de-kube-api-access-9bhr9" seLinuxMountContext=""
Oct 29 21:46:19.022723 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020594    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="259e6b95-d64c-4101-871f-4a933d4ac38f" volumeName="kubernetes.io/configmap/259e6b95-d64c-4101-871f-4a933d4ac38f-ovnkube-script-lib" seLinuxMountContext=""
Oct 29 21:46:19.022723 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020603    4962 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72352328-6f28-47ae-9190-0176cedaed4a" volumeName="kubernetes.io/configmap/72352328-6f28-47ae-9190-0176cedaed4a-nginx-conf" seLinuxMountContext=""
Oct 29 21:46:19.022723 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020613    4962 reconstruct.go:97] "Volume reconstruction finished"
Oct 29 21:46:19.022723 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.020618    4962 reconciler.go:29] "Reconciler: start to sync state"
Oct 29 21:46:19.039666 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.039647    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/1.log"
Oct 29 21:46:19.039775 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.039678    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/1.log"
Oct 29 21:46:19.046667 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.046611    4962 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 29 21:46:19.049182 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.049156    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74bdq_259e6b95-d64c-4101-871f-4a933d4ac38f/ovn-acl-logging/0.log"
Oct 29 21:46:19.049479 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.049457    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74bdq_259e6b95-d64c-4101-871f-4a933d4ac38f/ovn-acl-logging/0.log"
Oct 29 21:46:19.049713 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.049694    4962 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 29 21:46:19.049713 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.049716    4962 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 29 21:46:19.049854 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.049736    4962 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 29 21:46:19.049854 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:19.049775    4962 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 29 21:46:19.059699 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.059661    4962 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 29 21:46:19.059797 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.059738    4962 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:46:19.060005 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.059981    4962 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:46:19.066336 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.066310    4962 generic.go:356] "Generic (PLEG): container finished" podID="9400c4a1-ccf5-4b45-94f8-e37079762e4f" containerID="6f3433933e446ad2be68d92dd40adf255d899677bb24c1a56cb2e75de77353ae" exitCode=0
Oct 29 21:46:19.066336 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.066336    4962 generic.go:356] "Generic (PLEG): container finished" podID="9400c4a1-ccf5-4b45-94f8-e37079762e4f" containerID="ac7376cc4ea396e59e08092e2d7e19493bf127bdde5432ef1193d35b7e13c657" exitCode=0
Oct 29 21:46:19.066451 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.066346    4962 generic.go:356] "Generic (PLEG): container finished" podID="9400c4a1-ccf5-4b45-94f8-e37079762e4f" containerID="3cc736966201b4831f6ae362e62bf7742d447e94ec442532bb26c1b35fd0559f" exitCode=0
Oct 29 21:46:19.066451 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.066352    4962 generic.go:356] "Generic (PLEG): container finished" podID="9400c4a1-ccf5-4b45-94f8-e37079762e4f" containerID="19ea69748f71f0d6efd89587b876593eee4fe7df45ca18bb8e082be12f5c1c1e" exitCode=0
Oct 29 21:46:19.066451 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.066357    4962 generic.go:356] "Generic (PLEG): container finished" podID="9400c4a1-ccf5-4b45-94f8-e37079762e4f" containerID="fc79f20e73ad002f644f41df19e1bb1f25667d0a086baf6a7f752d7523435cda" exitCode=0
Oct 29 21:46:19.066451 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.066362    4962 generic.go:356] "Generic (PLEG): container finished" podID="9400c4a1-ccf5-4b45-94f8-e37079762e4f" containerID="6e9dc9d73f2e678b39692bcf2a191d2d79352e50ccd4d322849a8e3fc6fbef2b" exitCode=0
Oct 29 21:46:19.071669 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.071647    4962 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:46:19.071803 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.071725    4962 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 21:46:19.071964 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.071940    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74bdq_259e6b95-d64c-4101-871f-4a933d4ac38f/ovn-acl-logging/0.log"
Oct 29 21:46:19.072255 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.072228    4962 generic.go:356] "Generic (PLEG): container finished" podID="259e6b95-d64c-4101-871f-4a933d4ac38f" containerID="e8d2e46a6f05d222f7e077887589b8e6a0b76ec5ae44079648d836889a8ddfae" exitCode=1
Oct 29 21:46:19.092192 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.092171    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/1.log"
Oct 29 21:46:19.092265 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.092204    4962 generic.go:356] "Generic (PLEG): container finished" podID="f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600" containerID="b727bfa5392f31848f4d9bdd2173e4c65b562208ae7f10f517776dd67c5132f8" exitCode=255
Oct 29 21:46:19.099802 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.099782    4962 generic.go:356] "Generic (PLEG): container finished" podID="493b35d9c8008ff0c00d95f4c3e7e953" containerID="d40fb390702c1e1a031beead78275082701bf35228ebb6151209a529a6aa3ffc" exitCode=0
Oct 29 21:46:19.112563 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.112513    4962 manager.go:324] Recovery completed
Oct 29 21:46:19.130282 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.130260    4962 cpu_manager.go:226] "Starting" policy="none"
Oct 29 21:46:19.130282 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.130277    4962 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 29 21:46:19.130429 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.130318    4962 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 29 21:46:19.130491 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.130475    4962 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 29 21:46:19.130540 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.130491    4962 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 29 21:46:19.130540 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.130515    4962 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 29 21:46:19.130540 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.130530    4962 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 29 21:46:19.130540 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.130537    4962 policy_none.go:50] "Start"
Oct 29 21:46:19.130728 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.130564    4962 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 29 21:46:19.130728 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.130577    4962 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 29 21:46:19.130813 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.130774    4962 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 29 21:46:19.130813 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.130783    4962 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 29 21:46:19.130813 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.130791    4962 policy_none.go:44] "Start"
Oct 29 21:46:19.135549 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.135524    4962 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 29 21:46:19.138229 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.138215    4962 manager.go:341] "Starting Device Plugin manager"
Oct 29 21:46:19.138886 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:19.138244    4962 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 29 21:46:19.138932 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.138893    4962 server.go:87] "Starting device plugin registration server"
Oct 29 21:46:19.139195 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.139183    4962 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 29 21:46:19.139241 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.139198    4962 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 29 21:46:19.139331 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.139310    4962 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 29 21:46:19.139460 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.139422    4962 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 29 21:46:19.139515 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.139469    4962 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 29 21:46:19.139515 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.139476    4962 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 29 21:46:19.139712 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.139527    4962 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T21:46:19.139458054Z","UUID":"87e6034c-b363-4753-86ab-9cd09710f822","Handler":null,"Name":"","Endpoint":""}
Oct 29 21:46:19.140886 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.140868    4962 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 29 21:46:19.140990 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:19.140885    4962 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 29 21:46:19.140990 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.140894    4962 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 29 21:46:19.149878 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.149834    4962 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal"]
Oct 29 21:46:19.150632 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.150615    4962 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal"
Oct 29 21:46:19.150794 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.150754    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal" event={"ID":"4db57e6b6465b77567369446b9e61568","Type":"ContainerStarted","Data":"5d1f40acaa9a7bef0f4497c34ad4ddd4cd7c646c42c9d1439579502bccb6c6d1"}
Oct 29 21:46:19.150864 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.150807    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal" event={"ID":"4db57e6b6465b77567369446b9e61568","Type":"ContainerStarted","Data":"1b7d7a6db961bfef862e90db2aca93842cf4863e6ab26c33af1b217ff4276fba"}
Oct 29 21:46:19.150864 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.150856    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal" event={"ID":"493b35d9c8008ff0c00d95f4c3e7e953","Type":"ContainerStarted","Data":"7b067d859195a198b899849ee55f0045da39c91e2aa9ab7b399e37e4a9e98d50"}
Oct 29 21:46:19.150976 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.150874    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal" event={"ID":"493b35d9c8008ff0c00d95f4c3e7e953","Type":"ContainerDied","Data":"d40fb390702c1e1a031beead78275082701bf35228ebb6151209a529a6aa3ffc"}
Oct 29 21:46:19.150976 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.150894    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal" event={"ID":"493b35d9c8008ff0c00d95f4c3e7e953","Type":"ContainerStarted","Data":"dfd9aba9499701d8616e24cc1e3b84b531f788c45a805468f92b6e2f9440ce3f"}
Oct 29 21:46:19.151075 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.151028    4962 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal"
Oct 29 21:46:19.174407 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.174390    4962 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 21:46:19.174475 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:19.174431    4962 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-146.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal"
Oct 29 21:46:19.175097 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.175083    4962 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 21:46:19.175139 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:19.175120    4962 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal"
Oct 29 21:46:19.239974 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.239952    4962 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 21:46:19.240995 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.240976    4962 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 21:46:19.241115 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.241005    4962 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 21:46:19.241115 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.241021    4962 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-146.ec2.internal" event="NodeHasSufficientPID"
Oct 29 21:46:19.241115 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.241069    4962 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-146.ec2.internal"
Oct 29 21:46:19.260522 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.260503    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4db57e6b6465b77567369446b9e61568-config\") pod \"kube-apiserver-proxy-ip-10-0-129-146.ec2.internal\" (UID: \"4db57e6b6465b77567369446b9e61568\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal"
Oct 29 21:46:19.260601 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.260526    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/493b35d9c8008ff0c00d95f4c3e7e953-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal\" (UID: \"493b35d9c8008ff0c00d95f4c3e7e953\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal"
Oct 29 21:46:19.260601 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.260544    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/493b35d9c8008ff0c00d95f4c3e7e953-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal\" (UID: \"493b35d9c8008ff0c00d95f4c3e7e953\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal"
Oct 29 21:46:19.275656 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.275642    4962 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-146.ec2.internal"
Oct 29 21:46:19.275728 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.275685    4962 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-146.ec2.internal"
Oct 29 21:46:19.361228 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.361205    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/493b35d9c8008ff0c00d95f4c3e7e953-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal\" (UID: \"493b35d9c8008ff0c00d95f4c3e7e953\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal"
Oct 29 21:46:19.361321 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.361231    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/493b35d9c8008ff0c00d95f4c3e7e953-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal\" (UID: \"493b35d9c8008ff0c00d95f4c3e7e953\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal"
Oct 29 21:46:19.361321 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.361248    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4db57e6b6465b77567369446b9e61568-config\") pod \"kube-apiserver-proxy-ip-10-0-129-146.ec2.internal\" (UID: \"4db57e6b6465b77567369446b9e61568\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal"
Oct 29 21:46:19.361321 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.361283    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4db57e6b6465b77567369446b9e61568-config\") pod \"kube-apiserver-proxy-ip-10-0-129-146.ec2.internal\" (UID: \"4db57e6b6465b77567369446b9e61568\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal"
Oct 29 21:46:19.361428 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.361332    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/493b35d9c8008ff0c00d95f4c3e7e953-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal\" (UID: \"493b35d9c8008ff0c00d95f4c3e7e953\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal"
Oct 29 21:46:19.361428 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.361332    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/493b35d9c8008ff0c00d95f4c3e7e953-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal\" (UID: \"493b35d9c8008ff0c00d95f4c3e7e953\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal"
Oct 29 21:46:19.989506 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:19.989480    4962 apiserver.go:52] "Watching apiserver"
Oct 29 21:46:20.040338 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.040315    4962 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 29 21:46:20.041058 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.041030    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-jtrf8","openshift-cluster-node-tuning-operator/tuned-s6mfd","openshift-network-diagnostics/network-check-source-6f78d59f99-wq26q","openshift-multus/multus-additional-cni-plugins-lzjtw","openshift-kube-storage-version-migrator/migrator-66df5d496b-gs6zr","openshift-image-registry/node-ca-knxtf","kube-system/kas-connection-checker-685895c7c9-lqjq8","openshift-dns/dns-default-swlfh","kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal","kube-system/global-pull-secret-syncer-hwq7z","openshift-ovn-kubernetes/ovnkube-node-74bdq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal","openshift-multus/network-metrics-daemon-l7tkk","openshift-console-operator/console-operator-7cd56b8dbc-jdps8","openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9","openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k","openshift-network-operator/iptables-alerter-t997r","kube-system/konnectivity-agent-g2rdv","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2","kube-system/kas-connection-checker-685895c7c9-fkzj6","openshift-network-diagnostics/network-check-target-pkmzk","openshift-ingress-canary/ingress-canary-dkzbq","openshift-network-console/networking-console-plugin-86c478bb88-9np86","openshift-dns/node-resolver-rrmxb","openshift-insights/insights-runtime-extractor-dtgwl","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cv94k","openshift-image-registry/image-registry-55df9ddfdb-9gj4j","kube-system/kas-connection-checker-685895c7c9-4w7w5","openshift-multus/multus-ts2kk","openshift-image-registry/image-registry-547f4fc48c-tpxdx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz","openshift-ingress/router-default-6d4df84b7f-nm9mn"]
Oct 29 21:46:20.042372 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.042347    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:20.042893 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.042878    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"
Oct 29 21:46:20.044964 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.044945    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:20.045804 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.045787    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:20.046260 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.046243    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-9np86"
Oct 29 21:46:20.054640 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.054609    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:46:20.056133 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.056113    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:20.056452 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.056435    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 29 21:46:20.056549 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.056536    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:46:20.056696 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.056680    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 29 21:46:20.056777 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.056764    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:20.058221 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.058201    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 29 21:46:20.058362 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.058344    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:20.059436 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.059421    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 29 21:46:20.059436 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.059422    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 29 21:46:20.059901 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.059887    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 29 21:46:20.059901 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.059895    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 29 21:46:20.060002 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.059895    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:46:20.060098 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.060085    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rh2k6\""
Oct 29 21:46:20.060149 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.060107    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:46:20.060149 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.060115    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 29 21:46:20.060149 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.060136    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-zphp7\""
Oct 29 21:46:20.060240 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.060139    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wm9k9\""
Oct 29 21:46:20.060240 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.060132    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 29 21:46:20.060443 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.060417    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 29 21:46:20.060443 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.060420    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-sxnlk\""
Oct 29 21:46:20.060615 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.060443    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 29 21:46:20.060615 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.060456    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-hx8mh\""
Oct 29 21:46:20.060615 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.060463    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 29 21:46:20.060615 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.060461    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 29 21:46:20.060615 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.060466    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 29 21:46:20.060615 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.060480    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 29 21:46:20.060860 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.060719    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 29 21:46:20.072255 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.072233    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 29 21:46:20.072367 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.072279    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:46:20.072367 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.072301    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:46:20.072367 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.072311    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 29 21:46:20.072367 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.072315    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 29 21:46:20.072367 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.072356    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 29 21:46:20.072367 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.072388    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 29 21:46:20.072657 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.072416    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 29 21:46:20.072657 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.072637    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 21:46:20.072657 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.072639    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 29 21:46:20.072908 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.072896    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 29 21:46:20.074670 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.074651    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 29 21:46:20.074753 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.074698    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 21:46:20.074753 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.074738    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 29 21:46:20.074753 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.074747    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 29 21:46:20.074753 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.074754    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 29 21:46:20.074917 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.074787    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 29 21:46:20.074917 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.074826    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:46:20.074917 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.074852    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 29 21:46:20.075272 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.075257    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 29 21:46:20.075986 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.075965    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 29 21:46:20.076062 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.076006    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 29 21:46:20.076188 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.076174    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 29 21:46:20.076262 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.076251    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jtcl2\""
Oct 29 21:46:20.076334 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.076266    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-hjsdq\""
Oct 29 21:46:20.076491 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.076477    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 29 21:46:20.076534 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.076495    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 29 21:46:20.078161 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.078136    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 29 21:46:20.078257 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.078168    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-pl8bq\""
Oct 29 21:46:20.078257 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.078177    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 29 21:46:20.078257 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.078224    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mlr8g\""
Oct 29 21:46:20.078257 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.078238    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-ldpp7\""
Oct 29 21:46:20.078257 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.078254    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8ld6j\""
Oct 29 21:46:20.079232 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.079200    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 29 21:46:20.079232 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.079204    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 29 21:46:20.079232 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.079226    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 29 21:46:20.079393 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.079227    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 29 21:46:20.079393 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.079230    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 29 21:46:20.079393 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.079210    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 29 21:46:20.079393 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.079261    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 29 21:46:20.079393 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.079275    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 29 21:46:20.079595 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.079578    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 29 21:46:20.079668 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.079640    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 29 21:46:20.080577 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.080557    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 29 21:46:20.080577 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.080560    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 29 21:46:20.080705 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.080583    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 29 21:46:20.080705 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.080609    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 29 21:46:20.080985 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.080965    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 29 21:46:20.081031 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.081003    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 29 21:46:20.081031 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.081016    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 29 21:46:20.083296 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.083272    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 29 21:46:20.083384 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.083368    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 29 21:46:20.083451 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.083435    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 29 21:46:20.083494 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.083440    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 29 21:46:20.083587 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.083572    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 29 21:46:20.083648 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.083601    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 29 21:46:20.083648 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.083602    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 29 21:46:20.083648 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.083625    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 29 21:46:20.083648 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.083632    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-p5fxq\""
Oct 29 21:46:20.083648 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.083631    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:46:20.083648 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.083645    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 29 21:46:20.083813 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.083654    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 29 21:46:20.083813 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.083647    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 29 21:46:20.083813 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.083808    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-znmmx\""
Oct 29 21:46:20.084038 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.084026    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 29 21:46:20.084110 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.084070    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 29 21:46:20.084110 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.084101    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-6chnm\""
Oct 29 21:46:20.084165 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.084124    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hzxgj\""
Oct 29 21:46:20.084165 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.084129    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jsffp\""
Oct 29 21:46:20.084165 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.084155    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hxxfk\""
Oct 29 21:46:20.084239 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.084210    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w9rdw\""
Oct 29 21:46:20.084239 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.084227    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-l6fj9\""
Oct 29 21:46:20.084352 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.084259    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ckcdt\""
Oct 29 21:46:20.084352 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.084258    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cgft7\""
Oct 29 21:46:20.084352 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.084260    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 29 21:46:20.084352 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.084270    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6jh7q\""
Oct 29 21:46:20.084352 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.084339    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7dqcg\""
Oct 29 21:46:20.084352 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.084349    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-t7vd6\""
Oct 29 21:46:20.084566 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.084363    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4sldf\""
Oct 29 21:46:20.084566 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.084389    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5fns5\""
Oct 29 21:46:20.084566 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.084431    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 21:46:20.084566 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.084501    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vdqjz\""
Oct 29 21:46:20.084566 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.084557    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 21:46:20.087922 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.087899    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 29 21:46:20.090225 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.090202    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 29 21:46:20.103975 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.103954    4962 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal"
Oct 29 21:46:20.104092 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.104080    4962 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal"
Oct 29 21:46:20.114681 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.114666    4962 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 29 21:46:20.127582 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.127566    4962 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 21:46:20.127633 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.127603    4962 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-146.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-146.ec2.internal"
Oct 29 21:46:20.128418 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.128406    4962 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 21:46:20.128454 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.128436    4962 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-146.ec2.internal"
Oct 29 21:46:20.164868 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.164849    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d45cab6-698a-4a61-be2c-58fe7d41c55c-crio-socket\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:20.164909 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.164877    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x75hw\" (UniqueName: \"kubernetes.io/projected/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-kube-api-access-x75hw\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.164909 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.164895    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-kubernetes\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.164988 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.164911    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af572bfb-940b-4226-b91f-d06554b40d3c-konnectivity-ca\") pod \"konnectivity-agent-g2rdv\" (UID: \"af572bfb-940b-4226-b91f-d06554b40d3c\") " pod="kube-system/konnectivity-agent-g2rdv"
Oct 29 21:46:20.164988 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.164927    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d45cab6-698a-4a61-be2c-58fe7d41c55c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:20.165078 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165006    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-cni-netd\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.165078 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165047    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whtg5\" (UniqueName: \"kubernetes.io/projected/1065c119-4756-47ad-960c-c99ee1fae963-kube-api-access-whtg5\") pod \"kas-connection-checker-685895c7c9-fkzj6\" (UID: \"1065c119-4756-47ad-960c-c99ee1fae963\") " pod="kube-system/kas-connection-checker-685895c7c9-fkzj6"
Oct 29 21:46:20.165167 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165079    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-hostroot\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.165167 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165104    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-installation-pull-secrets\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:20.165167 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165130    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69xwz\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-kube-api-access-69xwz\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:20.165308 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165245    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1953d5bb-b35f-4c91-a263-a3664cf7fab2-host-slash\") pod \"iptables-alerter-t997r\" (UID: \"1953d5bb-b35f-4c91-a263-a3664cf7fab2\") " pod="openshift-network-operator/iptables-alerter-t997r"
Oct 29 21:46:20.165308 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165270    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-run-openvswitch\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.165416 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165308    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af572bfb-940b-4226-b91f-d06554b40d3c-konnectivity-ca\") pod \"konnectivity-agent-g2rdv\" (UID: \"af572bfb-940b-4226-b91f-d06554b40d3c\") " pod="kube-system/konnectivity-agent-g2rdv"
Oct 29 21:46:20.165416 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165313    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9400c4a1-ccf5-4b45-94f8-e37079762e4f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:46:20.165416 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165363    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-installation-pull-secrets\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:20.165416 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165394    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbk9p\" (UniqueName: \"kubernetes.io/projected/9400c4a1-ccf5-4b45-94f8-e37079762e4f-kube-api-access-pbk9p\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:46:20.165600 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165423    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-var-lib-kubelet\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.165600 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165452    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/30af126f-08a0-49e2-9988-50b2f8380cfe-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2\" (UID: \"30af126f-08a0-49e2-9988-50b2f8380cfe\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2"
Oct 29 21:46:20.165600 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165476    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nbmt9\" (UID: \"0388d722-6c0b-4cf0-9f0f-7ee369004008\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:20.165600 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165502    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-cni-binary-copy\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.165600 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165526    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:46:20.165600 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165552    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99c9881d-05b3-4953-b484-3ce51520b84a-metrics-tls\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:20.165600 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165591    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/676904dc-4d4f-4bd6-b380-357d5d9a29d4-kubelet-config\") pod \"global-pull-secret-syncer-hwq7z\" (UID: \"676904dc-4d4f-4bd6-b380-357d5d9a29d4\") " pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:46:20.165792 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165625    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/30af126f-08a0-49e2-9988-50b2f8380cfe-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2\" (UID: \"30af126f-08a0-49e2-9988-50b2f8380cfe\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2"
Oct 29 21:46:20.165792 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165627    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-var-lib-cni-multus\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.165792 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165668    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs\") pod \"network-metrics-daemon-l7tkk\" (UID: \"0febea83-955d-4a56-8bc4-c883a4141109\") " pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:46:20.165792 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165704    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-log-socket\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.165792 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165734    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtz7z\" (UniqueName: \"kubernetes.io/projected/1eb2f410-512d-4e3e-838a-87c1e3b47207-kube-api-access-jtz7z\") pod \"cluster-monitoring-operator-884576c57-6p66l\" (UID: \"1eb2f410-512d-4e3e-838a-87c1e3b47207\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"
Oct 29 21:46:20.165792 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165757    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-tmp\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:20.165956 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165820    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-tmp\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:20.165956 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165862    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8428734b-4a9f-4df9-ac06-81074198a9ba-serviceca\") pod \"node-ca-knxtf\" (UID: \"8428734b-4a9f-4df9-ac06-81074198a9ba\") " pod="openshift-image-registry/node-ca-knxtf"
Oct 29 21:46:20.165956 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165885    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-cni-binary-copy\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.165956 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165889    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-run\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.165956 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165905    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-node-log\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.166179 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.165965    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-nbmt9\" (UID: \"0388d722-6c0b-4cf0-9f0f-7ee369004008\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:20.166179 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166004    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-serving-cert\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:20.166179 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166042    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hs6g\" (UniqueName: \"kubernetes.io/projected/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-kube-api-access-8hs6g\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:20.166179 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166063    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bhr9\" (UniqueName: \"kubernetes.io/projected/fc992647-89dd-44cc-ba19-3fdb03e125de-kube-api-access-9bhr9\") pod \"node-resolver-rrmxb\" (UID: \"fc992647-89dd-44cc-ba19-3fdb03e125de\") " pod="openshift-dns/node-resolver-rrmxb"
Oct 29 21:46:20.166179 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166088    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-var-lib-kubelet\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.166179 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166111    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-registration-dir\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:46:20.166179 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166135    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-kubelet\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.166179 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166141    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-nbmt9\" (UID: \"0388d722-6c0b-4cf0-9f0f-7ee369004008\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:20.166179 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166144    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8428734b-4a9f-4df9-ac06-81074198a9ba-serviceca\") pod \"node-ca-knxtf\" (UID: \"8428734b-4a9f-4df9-ac06-81074198a9ba\") " pod="openshift-image-registry/node-ca-knxtf"
Oct 29 21:46:20.166468 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166190    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5mps\" (UniqueName: \"kubernetes.io/projected/259e6b95-d64c-4101-871f-4a933d4ac38f-kube-api-access-v5mps\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.166468 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166194    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-serving-cert\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:20.166468 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166217    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-multus-socket-dir-parent\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.166468 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166323    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8428734b-4a9f-4df9-ac06-81074198a9ba-host\") pod \"node-ca-knxtf\" (UID: \"8428734b-4a9f-4df9-ac06-81074198a9ba\") " pod="openshift-image-registry/node-ca-knxtf"
Oct 29 21:46:20.166468 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166343    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4b5fdb27-5000-4cf8-a025-13ae5389d17f-image-registry-private-configuration\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:20.166468 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166364    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/259e6b95-d64c-4101-871f-4a933d4ac38f-ovn-node-metrics-cert\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.166468 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166408    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/676904dc-4d4f-4bd6-b380-357d5d9a29d4-original-pull-secret\") pod \"global-pull-secret-syncer-hwq7z\" (UID: \"676904dc-4d4f-4bd6-b380-357d5d9a29d4\") " pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:46:20.166468 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166426    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/676904dc-4d4f-4bd6-b380-357d5d9a29d4-dbus\") pod \"global-pull-secret-syncer-hwq7z\" (UID: \"676904dc-4d4f-4bd6-b380-357d5d9a29d4\") " pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:46:20.166468 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166452    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:20.166807 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166475    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-metrics-certs\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:20.166807 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166510    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-sys-fs\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:46:20.166807 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166537    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc992647-89dd-44cc-ba19-3fdb03e125de-hosts-file\") pod \"node-resolver-rrmxb\" (UID: \"fc992647-89dd-44cc-ba19-3fdb03e125de\") " pod="openshift-dns/node-resolver-rrmxb"
Oct 29 21:46:20.166807 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166544    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4b5fdb27-5000-4cf8-a025-13ae5389d17f-image-registry-private-configuration\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:20.166807 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166560    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-run-netns\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.166807 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166593    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/259e6b95-d64c-4101-871f-4a933d4ac38f-ovn-node-metrics-cert\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.166807 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166612    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-run-multus-certs\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.166807 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166654    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-var-lib-openvswitch\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.166807 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166684    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/676904dc-4d4f-4bd6-b380-357d5d9a29d4-original-pull-secret\") pod \"global-pull-secret-syncer-hwq7z\" (UID: \"676904dc-4d4f-4bd6-b380-357d5d9a29d4\") " pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:46:20.166807 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166700    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1eb2f410-512d-4e3e-838a-87c1e3b47207-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6p66l\" (UID: \"1eb2f410-512d-4e3e-838a-87c1e3b47207\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"
Oct 29 21:46:20.166807 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166728    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z46kb\" (UniqueName: \"kubernetes.io/projected/99c9881d-05b3-4953-b484-3ce51520b84a-kube-api-access-z46kb\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:20.166807 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166773    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-tuned\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.167218 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166870    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxjln\" (UniqueName: \"kubernetes.io/projected/07a537c1-669a-4a02-bc1f-f09ea24e47ca-kube-api-access-qxjln\") pod \"ingress-canary-dkzbq\" (UID: \"07a537c1-669a-4a02-bc1f-f09ea24e47ca\") " pod="openshift-ingress-canary/ingress-canary-dkzbq"
Oct 29 21:46:20.167218 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166890    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-tuned\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.167218 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166896    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1eb2f410-512d-4e3e-838a-87c1e3b47207-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-6p66l\" (UID: \"1eb2f410-512d-4e3e-838a-87c1e3b47207\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"
Oct 29 21:46:20.167218 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.166930    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9400c4a1-ccf5-4b45-94f8-e37079762e4f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:46:20.167218 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167047    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thktc\" (UniqueName: \"kubernetes.io/projected/30af126f-08a0-49e2-9988-50b2f8380cfe-kube-api-access-thktc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2\" (UID: \"30af126f-08a0-49e2-9988-50b2f8380cfe\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2"
Oct 29 21:46:20.167218 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167189    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:20.167218 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167204    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9400c4a1-ccf5-4b45-94f8-e37079762e4f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:46:20.167550 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167236    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:20.167550 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167326    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600-serving-cert\") pod \"console-operator-7cd56b8dbc-jdps8\" (UID: \"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:20.167550 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167403    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d45cab6-698a-4a61-be2c-58fe7d41c55c-data-volume\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:20.167550 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167456    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600-serving-cert\") pod \"console-operator-7cd56b8dbc-jdps8\" (UID: \"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:20.167550 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167463    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d45cab6-698a-4a61-be2c-58fe7d41c55c-data-volume\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:20.167550 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167472    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:20.167550 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167489    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-multus-daemon-config\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.167550 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167518    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-etc-selinux\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:46:20.167550 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167545    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b546ae66-9568-4840-a4e5-659d93eb80c5-tmp\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.167909 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167572    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1953d5bb-b35f-4c91-a263-a3664cf7fab2-iptables-alerter-script\") pod \"iptables-alerter-t997r\" (UID: \"1953d5bb-b35f-4c91-a263-a3664cf7fab2\") " pod="openshift-network-operator/iptables-alerter-t997r"
Oct 29 21:46:20.167909 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167591    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-multus-daemon-config\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.167909 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167621    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kr4j\" (UniqueName: \"kubernetes.io/projected/1953d5bb-b35f-4c91-a263-a3664cf7fab2-kube-api-access-2kr4j\") pod \"iptables-alerter-t997r\" (UID: \"1953d5bb-b35f-4c91-a263-a3664cf7fab2\") " pod="openshift-network-operator/iptables-alerter-t997r"
Oct 29 21:46:20.167909 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167710    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-run-ovn\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.167909 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167725    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-cnibin\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.167909 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167730    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b546ae66-9568-4840-a4e5-659d93eb80c5-tmp\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.167909 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167740    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngw2c\" (UniqueName: \"kubernetes.io/projected/0febea83-955d-4a56-8bc4-c883a4141109-kube-api-access-ngw2c\") pod \"network-metrics-daemon-l7tkk\" (UID: \"0febea83-955d-4a56-8bc4-c883a4141109\") " pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:46:20.167909 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167769    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-etc-openvswitch\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.167909 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167803    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1953d5bb-b35f-4c91-a263-a3664cf7fab2-iptables-alerter-script\") pod \"iptables-alerter-t997r\" (UID: \"1953d5bb-b35f-4c91-a263-a3664cf7fab2\") " pod="openshift-network-operator/iptables-alerter-t997r"
Oct 29 21:46:20.167909 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167816    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/259e6b95-d64c-4101-871f-4a933d4ac38f-ovnkube-script-lib\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.167909 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167858    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-etc-kubernetes\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.167909 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167885    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-image-registry-private-configuration\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:20.168382 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167917    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-stats-auth\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:20.168382 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167944    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqh8q\" (UniqueName: \"kubernetes.io/projected/a2536550-3bec-4756-b872-7b14a8eb78c3-kube-api-access-xqh8q\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:20.168382 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.167973    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-modprobe-d\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.168382 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168001    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-run-systemd\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.168382 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168027    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9400c4a1-ccf5-4b45-94f8-e37079762e4f-cnibin\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:46:20.168382 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168083    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-sysconfig\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.168382 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168082    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-image-registry-private-configuration\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:20.168382 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168085    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/259e6b95-d64c-4101-871f-4a933d4ac38f-ovnkube-script-lib\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.168382 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168120    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-sys\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.168382 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168147    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-stats-auth\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:20.168382 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168158    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-host\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.168382 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168189    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-slash\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.168382 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168229    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-run-ovn-kubernetes\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.168382 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168261    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0e7f3fff-da9c-4189-bd53-5d6586ee0a4c-config\") pod \"service-ca-operator-544d749c8c-d7x4k\" (UID: \"0e7f3fff-da9c-4189-bd53-5d6586ee0a4c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k"
Oct 29 21:46:20.168382 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168323    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm2gz\" (UniqueName: \"kubernetes.io/projected/0e7f3fff-da9c-4189-bd53-5d6586ee0a4c-kube-api-access-dm2gz\") pod \"service-ca-operator-544d749c8c-d7x4k\" (UID: \"0e7f3fff-da9c-4189-bd53-5d6586ee0a4c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k"
Oct 29 21:46:20.168937 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168396    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600-trusted-ca\") pod \"console-operator-7cd56b8dbc-jdps8\" (UID: \"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:20.168937 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168454    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0e7f3fff-da9c-4189-bd53-5d6586ee0a4c-config\") pod \"service-ca-operator-544d749c8c-d7x4k\" (UID: \"0e7f3fff-da9c-4189-bd53-5d6586ee0a4c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k"
Oct 29 21:46:20.168937 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168476    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-multus-cni-dir\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.168937 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168568    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-run-k8s-cni-cncf-io\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.168937 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168598    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9400c4a1-ccf5-4b45-94f8-e37079762e4f-cni-binary-copy\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:46:20.168937 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168614    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-lib-modules\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.168937 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168650    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600-trusted-ca\") pod \"console-operator-7cd56b8dbc-jdps8\" (UID: \"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:20.168937 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168644    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mppj9\" (UniqueName: \"kubernetes.io/projected/8d45cab6-698a-4a61-be2c-58fe7d41c55c-kube-api-access-mppj9\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:20.168937 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168716    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-cni-bin\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.168937 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168744    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.168937 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168762    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9400c4a1-ccf5-4b45-94f8-e37079762e4f-cni-binary-copy\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:46:20.168937 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168772    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/259e6b95-d64c-4101-871f-4a933d4ac38f-ovnkube-config\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.168937 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168800    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52fgx\" (UniqueName: \"kubernetes.io/projected/5c0cf3cb-ed1a-453c-a3a4-9289b557ae3d-kube-api-access-52fgx\") pod \"migrator-66df5d496b-gs6zr\" (UID: \"5c0cf3cb-ed1a-453c-a3a4-9289b557ae3d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gs6zr"
Oct 29 21:46:20.168937 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168931    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv58t\" (UniqueName: \"kubernetes.io/projected/0816e7fb-46c8-49d0-9c07-36fc49188b84-kube-api-access-lv58t\") pod \"network-check-source-6f78d59f99-wq26q\" (UID: \"0816e7fb-46c8-49d0-9c07-36fc49188b84\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wq26q"
Oct 29 21:46:20.169567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.168974    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/259e6b95-d64c-4101-871f-4a933d4ac38f-ovnkube-config\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.169567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169026    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6p66l\" (UID: \"1eb2f410-512d-4e3e-838a-87c1e3b47207\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"
Oct 29 21:46:20.169567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169047    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07a537c1-669a-4a02-bc1f-f09ea24e47ca-cert\") pod \"ingress-canary-dkzbq\" (UID: \"07a537c1-669a-4a02-bc1f-f09ea24e47ca\") " pod="openshift-ingress-canary/ingress-canary-dkzbq"
Oct 29 21:46:20.169567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169070    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfnx5\" (UniqueName: \"kubernetes.io/projected/8428734b-4a9f-4df9-ac06-81074198a9ba-kube-api-access-vfnx5\") pod \"node-ca-knxtf\" (UID: \"8428734b-4a9f-4df9-ac06-81074198a9ba\") " pod="openshift-image-registry/node-ca-knxtf"
Oct 29 21:46:20.169567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169096    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0e7f3fff-da9c-4189-bd53-5d6586ee0a4c-serving-cert\") pod \"service-ca-operator-544d749c8c-d7x4k\" (UID: \"0e7f3fff-da9c-4189-bd53-5d6586ee0a4c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k"
Oct 29 21:46:20.169567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169119    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-ca-trust-extracted\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:20.169567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169151    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-bound-sa-token\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:20.169567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169176    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/259e6b95-d64c-4101-871f-4a933d4ac38f-env-overrides\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.169567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169265    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07a537c1-669a-4a02-bc1f-f09ea24e47ca-cert\") pod \"ingress-canary-dkzbq\" (UID: \"07a537c1-669a-4a02-bc1f-f09ea24e47ca\") " pod="openshift-ingress-canary/ingress-canary-dkzbq"
Oct 29 21:46:20.169567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169319    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62lfn\" (UniqueName: \"kubernetes.io/projected/ce551e36-0249-4723-b814-a98f0a282eed-kube-api-access-62lfn\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:46:20.169567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169330    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-ca-trust-extracted\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:20.169567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169347    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/259e6b95-d64c-4101-871f-4a933d4ac38f-env-overrides\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.169567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169354    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc992647-89dd-44cc-ba19-3fdb03e125de-tmp-dir\") pod \"node-resolver-rrmxb\" (UID: \"fc992647-89dd-44cc-ba19-3fdb03e125de\") " pod="openshift-dns/node-resolver-rrmxb"
Oct 29 21:46:20.169567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169383    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-default-certificate\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:20.169567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169400    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9400c4a1-ccf5-4b45-94f8-e37079762e4f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:46:20.169567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169423    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c9881d-05b3-4953-b484-3ce51520b84a-tmp-dir\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:20.169567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169427    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc992647-89dd-44cc-ba19-3fdb03e125de-tmp-dir\") pod \"node-resolver-rrmxb\" (UID: \"fc992647-89dd-44cc-ba19-3fdb03e125de\") " pod="openshift-dns/node-resolver-rrmxb"
Oct 29 21:46:20.169567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169439    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-sysctl-d\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.169567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169477    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0e7f3fff-da9c-4189-bd53-5d6586ee0a4c-serving-cert\") pod \"service-ca-operator-544d749c8c-d7x4k\" (UID: \"0e7f3fff-da9c-4189-bd53-5d6586ee0a4c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k"
Oct 29 21:46:20.169567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169477    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/72352328-6f28-47ae-9190-0176cedaed4a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9np86\" (UID: \"72352328-6f28-47ae-9190-0176cedaed4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9np86"
Oct 29 21:46:20.169567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169508    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4b5fdb27-5000-4cf8-a025-13ae5389d17f-installation-pull-secrets\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:20.169567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169523    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99c9881d-05b3-4953-b484-3ce51520b84a-tmp-dir\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:20.169567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169548    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/99c9881d-05b3-4953-b484-3ce51520b84a-config-volume\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169605    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-default-certificate\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169641    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9400c4a1-ccf5-4b45-94f8-e37079762e4f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169643    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9np86\" (UID: \"72352328-6f28-47ae-9190-0176cedaed4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9np86"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169667    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/72352328-6f28-47ae-9190-0176cedaed4a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-9np86\" (UID: \"72352328-6f28-47ae-9190-0176cedaed4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9np86"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169687    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4b5fdb27-5000-4cf8-a025-13ae5389d17f-installation-pull-secrets\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169696    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169729    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m29ck\" (UniqueName: \"kubernetes.io/projected/0388d722-6c0b-4cf0-9f0f-7ee369004008-kube-api-access-m29ck\") pod \"cluster-samples-operator-6ddb6c8b94-nbmt9\" (UID: \"0388d722-6c0b-4cf0-9f0f-7ee369004008\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169752    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-snapshots\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169759    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/99c9881d-05b3-4953-b484-3ce51520b84a-config-volume\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169856    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-multus-conf-dir\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169886    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-snapshots\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169918    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4b5fdb27-5000-4cf8-a025-13ae5389d17f-ca-trust-extracted\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169957    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.169989    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/30af126f-08a0-49e2-9988-50b2f8380cfe-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2\" (UID: \"30af126f-08a0-49e2-9988-50b2f8380cfe\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170004    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4b5fdb27-5000-4cf8-a025-13ae5389d17f-ca-trust-extracted\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170023    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-service-ca-bundle\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170062    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-var-lib-cni-bin\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170079    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9400c4a1-ccf5-4b45-94f8-e37079762e4f-system-cni-dir\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170122    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/30af126f-08a0-49e2-9988-50b2f8380cfe-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2\" (UID: \"30af126f-08a0-49e2-9988-50b2f8380cfe\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170168    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-systemd\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170185    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-service-ca-bundle\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170211    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-certificates\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170228    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-systemd-units\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170244    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xntk2\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-kube-api-access-xntk2\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:20.170314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170260    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-socket-dir\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:46:20.171150 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170332    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b5fdb27-5000-4cf8-a025-13ae5389d17f-trusted-ca\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:20.171150 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170344    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-certificates\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:20.171150 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170363    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brg9z\" (UniqueName: \"kubernetes.io/projected/f934ff4f-dadc-469b-a87e-16db95e3181f-kube-api-access-brg9z\") pod \"network-check-target-pkmzk\" (UID: \"f934ff4f-dadc-469b-a87e-16db95e3181f\") " pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:46:20.171150 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170391    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-certificates\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:20.171150 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170417    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-trusted-ca\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:20.171150 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170439    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-bound-sa-token\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:20.171150 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170466    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-sysctl-conf\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.171150 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170587    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2ml4\" (UniqueName: \"kubernetes.io/projected/530bea30-e132-4b1a-9c6d-e421c216060a-kube-api-access-d2ml4\") pod \"volume-data-source-validator-58c9f5f46d-cv94k\" (UID: \"530bea30-e132-4b1a-9c6d-e421c216060a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cv94k"
Oct 29 21:46:20.171150 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170595    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-certificates\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:20.171150 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170610    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b5fdb27-5000-4cf8-a025-13ae5389d17f-trusted-ca\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:20.171150 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170619    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9z57\" (UniqueName: \"kubernetes.io/projected/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600-kube-api-access-v9z57\") pod \"console-operator-7cd56b8dbc-jdps8\" (UID: \"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:20.171150 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170649    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-system-cni-dir\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.171150 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170674    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-run-netns\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.171150 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170697    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-device-dir\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:46:20.171150 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170739    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600-config\") pod \"console-operator-7cd56b8dbc-jdps8\" (UID: \"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:20.171150 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170765    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-trusted-ca\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:20.171150 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170769    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af572bfb-940b-4226-b91f-d06554b40d3c-agent-certs\") pod \"konnectivity-agent-g2rdv\" (UID: \"af572bfb-940b-4226-b91f-d06554b40d3c\") " pod="kube-system/konnectivity-agent-g2rdv"
Oct 29 21:46:20.171150 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170885    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxt72\" (UniqueName: \"kubernetes.io/projected/73358782-c31a-4cb4-a0c2-377a1bceae4e-kube-api-access-sxt72\") pod \"kas-connection-checker-685895c7c9-lqjq8\" (UID: \"73358782-c31a-4cb4-a0c2-377a1bceae4e\") " pod="kube-system/kas-connection-checker-685895c7c9-lqjq8"
Oct 29 21:46:20.171150 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170895    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600-config\") pod \"console-operator-7cd56b8dbc-jdps8\" (UID: \"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:20.171150 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170912    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7dvt\" (UniqueName: \"kubernetes.io/projected/b546ae66-9568-4840-a4e5-659d93eb80c5-kube-api-access-x7dvt\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.171150 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.170998    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kmlt\" (UniqueName: \"kubernetes.io/projected/d172a43b-a20c-4fc3-a1cc-77a697312061-kube-api-access-8kmlt\") pod \"kas-connection-checker-685895c7c9-4w7w5\" (UID: \"d172a43b-a20c-4fc3-a1cc-77a697312061\") " pod="kube-system/kas-connection-checker-685895c7c9-4w7w5"
Oct 29 21:46:20.171150 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.171010    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af572bfb-940b-4226-b91f-d06554b40d3c-agent-certs\") pod \"konnectivity-agent-g2rdv\" (UID: \"af572bfb-940b-4226-b91f-d06554b40d3c\") " pod="kube-system/konnectivity-agent-g2rdv"
Oct 29 21:46:20.186912 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.186887    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtz7z\" (UniqueName: \"kubernetes.io/projected/1eb2f410-512d-4e3e-838a-87c1e3b47207-kube-api-access-jtz7z\") pod \"cluster-monitoring-operator-884576c57-6p66l\" (UID: \"1eb2f410-512d-4e3e-838a-87c1e3b47207\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"
Oct 29 21:46:20.190999 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.190983    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbk9p\" (UniqueName: \"kubernetes.io/projected/9400c4a1-ccf5-4b45-94f8-e37079762e4f-kube-api-access-pbk9p\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:46:20.191770 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.191751    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69xwz\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-kube-api-access-69xwz\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:20.195338 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.195320    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whtg5\" (UniqueName: \"kubernetes.io/projected/1065c119-4756-47ad-960c-c99ee1fae963-kube-api-access-whtg5\") pod \"kas-connection-checker-685895c7c9-fkzj6\" (UID: \"1065c119-4756-47ad-960c-c99ee1fae963\") " pod="kube-system/kas-connection-checker-685895c7c9-fkzj6"
Oct 29 21:46:20.195510 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.195492    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x75hw\" (UniqueName: \"kubernetes.io/projected/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-kube-api-access-x75hw\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.195605 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.195592    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hs6g\" (UniqueName: \"kubernetes.io/projected/55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97-kube-api-access-8hs6g\") pod \"insights-operator-7c4986d449-jtrf8\" (UID: \"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97\") " pod="openshift-insights/insights-operator-7c4986d449-jtrf8"
Oct 29 21:46:20.195656 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.195637    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bhr9\" (UniqueName: \"kubernetes.io/projected/fc992647-89dd-44cc-ba19-3fdb03e125de-kube-api-access-9bhr9\") pod \"node-resolver-rrmxb\" (UID: \"fc992647-89dd-44cc-ba19-3fdb03e125de\") " pod="openshift-dns/node-resolver-rrmxb"
Oct 29 21:46:20.195699 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.195690    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5mps\" (UniqueName: \"kubernetes.io/projected/259e6b95-d64c-4101-871f-4a933d4ac38f-kube-api-access-v5mps\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.200529 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.200512    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52fgx\" (UniqueName: \"kubernetes.io/projected/5c0cf3cb-ed1a-453c-a3a4-9289b557ae3d-kube-api-access-52fgx\") pod \"migrator-66df5d496b-gs6zr\" (UID: \"5c0cf3cb-ed1a-453c-a3a4-9289b557ae3d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-gs6zr"
Oct 29 21:46:20.200608 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.200525    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngw2c\" (UniqueName: \"kubernetes.io/projected/0febea83-955d-4a56-8bc4-c883a4141109-kube-api-access-ngw2c\") pod \"network-metrics-daemon-l7tkk\" (UID: \"0febea83-955d-4a56-8bc4-c883a4141109\") " pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:46:20.200608 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.200592    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv58t\" (UniqueName: \"kubernetes.io/projected/0816e7fb-46c8-49d0-9c07-36fc49188b84-kube-api-access-lv58t\") pod \"network-check-source-6f78d59f99-wq26q\" (UID: \"0816e7fb-46c8-49d0-9c07-36fc49188b84\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wq26q"
Oct 29 21:46:20.200701 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.200687    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z46kb\" (UniqueName: \"kubernetes.io/projected/99c9881d-05b3-4953-b484-3ce51520b84a-kube-api-access-z46kb\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:20.205823 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.205807    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqh8q\" (UniqueName: \"kubernetes.io/projected/a2536550-3bec-4756-b872-7b14a8eb78c3-kube-api-access-xqh8q\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:20.206381 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.206355    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mppj9\" (UniqueName: \"kubernetes.io/projected/8d45cab6-698a-4a61-be2c-58fe7d41c55c-kube-api-access-mppj9\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:20.206469 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.206439    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-bound-sa-token\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:20.206725 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.206701    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm2gz\" (UniqueName: \"kubernetes.io/projected/0e7f3fff-da9c-4189-bd53-5d6586ee0a4c-kube-api-access-dm2gz\") pod \"service-ca-operator-544d749c8c-d7x4k\" (UID: \"0e7f3fff-da9c-4189-bd53-5d6586ee0a4c\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k"
Oct 29 21:46:20.206802 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.206719    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfnx5\" (UniqueName: \"kubernetes.io/projected/8428734b-4a9f-4df9-ac06-81074198a9ba-kube-api-access-vfnx5\") pod \"node-ca-knxtf\" (UID: \"8428734b-4a9f-4df9-ac06-81074198a9ba\") " pod="openshift-image-registry/node-ca-knxtf"
Oct 29 21:46:20.206802 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.206726    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62lfn\" (UniqueName: \"kubernetes.io/projected/ce551e36-0249-4723-b814-a98f0a282eed-kube-api-access-62lfn\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:46:20.206802 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.206723    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kr4j\" (UniqueName: \"kubernetes.io/projected/1953d5bb-b35f-4c91-a263-a3664cf7fab2-kube-api-access-2kr4j\") pod \"iptables-alerter-t997r\" (UID: \"1953d5bb-b35f-4c91-a263-a3664cf7fab2\") " pod="openshift-network-operator/iptables-alerter-t997r"
Oct 29 21:46:20.206802 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.206756    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thktc\" (UniqueName: \"kubernetes.io/projected/30af126f-08a0-49e2-9988-50b2f8380cfe-kube-api-access-thktc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2\" (UID: \"30af126f-08a0-49e2-9988-50b2f8380cfe\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2"
Oct 29 21:46:20.206933 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.206859    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9z57\" (UniqueName: \"kubernetes.io/projected/f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600-kube-api-access-v9z57\") pod \"console-operator-7cd56b8dbc-jdps8\" (UID: \"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:20.207210 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.207190    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2ml4\" (UniqueName: \"kubernetes.io/projected/530bea30-e132-4b1a-9c6d-e421c216060a-kube-api-access-d2ml4\") pod \"volume-data-source-validator-58c9f5f46d-cv94k\" (UID: \"530bea30-e132-4b1a-9c6d-e421c216060a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cv94k"
Oct 29 21:46:20.207411 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.207396    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxjln\" (UniqueName: \"kubernetes.io/projected/07a537c1-669a-4a02-bc1f-f09ea24e47ca-kube-api-access-qxjln\") pod \"ingress-canary-dkzbq\" (UID: \"07a537c1-669a-4a02-bc1f-f09ea24e47ca\") " pod="openshift-ingress-canary/ingress-canary-dkzbq"
Oct 29 21:46:20.210437 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.210421    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kmlt\" (UniqueName: \"kubernetes.io/projected/d172a43b-a20c-4fc3-a1cc-77a697312061-kube-api-access-8kmlt\") pod \"kas-connection-checker-685895c7c9-4w7w5\" (UID: \"d172a43b-a20c-4fc3-a1cc-77a697312061\") " pod="kube-system/kas-connection-checker-685895c7c9-4w7w5"
Oct 29 21:46:20.211861 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.211837    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-bound-sa-token\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:20.212775 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.212759    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m29ck\" (UniqueName: \"kubernetes.io/projected/0388d722-6c0b-4cf0-9f0f-7ee369004008-kube-api-access-m29ck\") pod \"cluster-samples-operator-6ddb6c8b94-nbmt9\" (UID: \"0388d722-6c0b-4cf0-9f0f-7ee369004008\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:20.215487 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.215471    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xntk2\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-kube-api-access-xntk2\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:20.217102 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.217080    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7dvt\" (UniqueName: \"kubernetes.io/projected/b546ae66-9568-4840-a4e5-659d93eb80c5-kube-api-access-x7dvt\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.217569 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.217549    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxt72\" (UniqueName: \"kubernetes.io/projected/73358782-c31a-4cb4-a0c2-377a1bceae4e-kube-api-access-sxt72\") pod \"kas-connection-checker-685895c7c9-lqjq8\" (UID: \"73358782-c31a-4cb4-a0c2-377a1bceae4e\") " pod="kube-system/kas-connection-checker-685895c7c9-lqjq8"
Oct 29 21:46:20.217739 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.217725    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brg9z\" (UniqueName: \"kubernetes.io/projected/f934ff4f-dadc-469b-a87e-16db95e3181f-kube-api-access-brg9z\") pod \"network-check-target-pkmzk\" (UID: \"f934ff4f-dadc-469b-a87e-16db95e3181f\") " pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:46:20.271785 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.271739    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-etc-kubernetes\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.271785 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.271764    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-modprobe-d\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.271938 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.271788    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-run-systemd\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.271938 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.271802    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9400c4a1-ccf5-4b45-94f8-e37079762e4f-cnibin\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:46:20.271938 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.271818    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-sysconfig\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.271938 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.271836    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-sys\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.271938 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.271841    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-etc-kubernetes\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.271938 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.271855    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-host\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.271938 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.271877    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-run-systemd\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.271938 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.271883    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9400c4a1-ccf5-4b45-94f8-e37079762e4f-cnibin\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:46:20.271938 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.271912    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-host\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.271938 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.271916    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-modprobe-d\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.271938 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.271934    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-sysconfig\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.272279 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.271948    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-sys\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.272279 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272003    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-slash\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.272352 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272338    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-slash\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.272398 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272364    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-run-ovn-kubernetes\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.272398 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272380    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-multus-cni-dir\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.272398 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272395    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-run-k8s-cni-cncf-io\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.272504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272408    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-lib-modules\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.272504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272423    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-cni-bin\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.272504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272437    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.272504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272453    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6p66l\" (UID: \"1eb2f410-512d-4e3e-838a-87c1e3b47207\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"
Oct 29 21:46:20.272504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272463    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-run-ovn-kubernetes\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.272504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272469    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-multus-cni-dir\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.272504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272474    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-sysctl-d\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.272504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272493    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-run-k8s-cni-cncf-io\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.272504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272501    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-cni-bin\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.272806 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272513    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9np86\" (UID: \"72352328-6f28-47ae-9190-0176cedaed4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9np86"
Oct 29 21:46:20.272806 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272516    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.272806 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272543    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:20.272806 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.272560    4962 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 29 21:46:20.272806 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272566    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-sysctl-d\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.272806 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272567    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-lib-modules\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.272806 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.272593    4962 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 29 21:46:20.272806 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.272618    4962 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 21:46:20.272806 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272560    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-multus-conf-dir\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.272806 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.272630    4962 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-547f4fc48c-tpxdx: secret "image-registry-tls" not found
Oct 29 21:46:20.272806 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272651    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-multus-conf-dir\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.272806 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.272671    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert podName:72352328-6f28-47ae-9190-0176cedaed4a nodeName:}" failed. No retries permitted until 2026-10-29 21:46:20.772638223 +0000 UTC m=+1.845595184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9np86" (UID: "72352328-6f28-47ae-9190-0176cedaed4a") : secret "networking-console-plugin-cert" not found
Oct 29 21:46:20.272806 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272690    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:20.272806 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.272705    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls podName:1eb2f410-512d-4e3e-838a-87c1e3b47207 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:20.772692038 +0000 UTC m=+1.845648980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6p66l" (UID: "1eb2f410-512d-4e3e-838a-87c1e3b47207") : secret "cluster-monitoring-operator-tls" not found
Oct 29 21:46:20.272806 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.272732    4962 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:20.272806 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.272738    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls podName:4b5fdb27-5000-4cf8-a025-13ae5389d17f nodeName:}" failed. No retries permitted until 2026-10-29 21:46:20.772720453 +0000 UTC m=+1.845677398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls") pod "image-registry-547f4fc48c-tpxdx" (UID: "4b5fdb27-5000-4cf8-a025-13ae5389d17f") : secret "image-registry-tls" not found
Oct 29 21:46:20.272806 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.272756    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls podName:8d45cab6-698a-4a61-be2c-58fe7d41c55c nodeName:}" failed. No retries permitted until 2026-10-29 21:46:20.77274853 +0000 UTC m=+1.845705503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtgwl" (UID: "8d45cab6-698a-4a61-be2c-58fe7d41c55c") : secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:20.272806 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272766    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-var-lib-cni-bin\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.272806 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272782    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9400c4a1-ccf5-4b45-94f8-e37079762e4f-system-cni-dir\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:46:20.272806 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272798    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-systemd\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.272806 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272824    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-systemd-units\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272841    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-socket-dir\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272851    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9400c4a1-ccf5-4b45-94f8-e37079762e4f-system-cni-dir\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272862    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-sysctl-conf\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272868    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-systemd\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272872    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-var-lib-cni-bin\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272881    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-systemd-units\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272893    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-system-cni-dir\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272928    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-system-cni-dir\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272933    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-socket-dir\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272939    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-run-netns\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272964    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-device-dir\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272972    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-run-netns\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272975    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-sysctl-conf\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.272992    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d45cab6-698a-4a61-be2c-58fe7d41c55c-crio-socket\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273013    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-kubernetes\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273026    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-device-dir\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273048    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d45cab6-698a-4a61-be2c-58fe7d41c55c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273074    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-cni-netd\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273086    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8d45cab6-698a-4a61-be2c-58fe7d41c55c-crio-socket\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273099    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-hostroot\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.273122    4962 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273128    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-cni-netd\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273101    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-etc-kubernetes\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273142    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1953d5bb-b35f-4c91-a263-a3664cf7fab2-host-slash\") pod \"iptables-alerter-t997r\" (UID: \"1953d5bb-b35f-4c91-a263-a3664cf7fab2\") " pod="openshift-network-operator/iptables-alerter-t997r"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273125    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-hostroot\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273179    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1953d5bb-b35f-4c91-a263-a3664cf7fab2-host-slash\") pod \"iptables-alerter-t997r\" (UID: \"1953d5bb-b35f-4c91-a263-a3664cf7fab2\") " pod="openshift-network-operator/iptables-alerter-t997r"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.273186    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d45cab6-698a-4a61-be2c-58fe7d41c55c-kube-rbac-proxy-cm podName:8d45cab6-698a-4a61-be2c-58fe7d41c55c nodeName:}" failed. No retries permitted until 2026-10-29 21:46:20.773171165 +0000 UTC m=+1.846128124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8d45cab6-698a-4a61-be2c-58fe7d41c55c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dtgwl" (UID: "8d45cab6-698a-4a61-be2c-58fe7d41c55c") : configmap "kube-rbac-proxy" not found
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273208    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-run-openvswitch\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273224    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9400c4a1-ccf5-4b45-94f8-e37079762e4f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273240    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-var-lib-kubelet\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273225    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-run-openvswitch\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273265    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nbmt9\" (UID: \"0388d722-6c0b-4cf0-9f0f-7ee369004008\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273302    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273312    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-var-lib-kubelet\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273327    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99c9881d-05b3-4953-b484-3ce51520b84a-metrics-tls\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273351    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/676904dc-4d4f-4bd6-b380-357d5d9a29d4-kubelet-config\") pod \"global-pull-secret-syncer-hwq7z\" (UID: \"676904dc-4d4f-4bd6-b380-357d5d9a29d4\") " pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273356    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.273362    4962 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273311    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9400c4a1-ccf5-4b45-94f8-e37079762e4f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lzjtw\" (UID: \"9400c4a1-ccf5-4b45-94f8-e37079762e4f\") " pod="openshift-multus/multus-additional-cni-plugins-lzjtw"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273372    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-var-lib-cni-multus\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273398    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs\") pod \"network-metrics-daemon-l7tkk\" (UID: \"0febea83-955d-4a56-8bc4-c883a4141109\") " pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273403    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/676904dc-4d4f-4bd6-b380-357d5d9a29d4-kubelet-config\") pod \"global-pull-secret-syncer-hwq7z\" (UID: \"676904dc-4d4f-4bd6-b380-357d5d9a29d4\") " pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.273407    4962 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273409    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-var-lib-cni-multus\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.273408    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls podName:0388d722-6c0b-4cf0-9f0f-7ee369004008 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:20.77339472 +0000 UTC m=+1.846351668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-nbmt9" (UID: "0388d722-6c0b-4cf0-9f0f-7ee369004008") : secret "samples-operator-tls" not found
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.273481    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99c9881d-05b3-4953-b484-3ce51520b84a-metrics-tls podName:99c9881d-05b3-4953-b484-3ce51520b84a nodeName:}" failed. No retries permitted until 2026-10-29 21:46:20.773467066 +0000 UTC m=+1.846424015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/99c9881d-05b3-4953-b484-3ce51520b84a-metrics-tls") pod "dns-default-swlfh" (UID: "99c9881d-05b3-4953-b484-3ce51520b84a") : secret "dns-default-metrics-tls" not found
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273499    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-log-socket\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273529    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-run\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273553    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-node-log\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273564    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-log-socket\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273581    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-var-lib-kubelet\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273604    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-registration-dir\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273615    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-node-log\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273627    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-kubelet\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273637    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-var-lib-kubelet\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273655    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-multus-socket-dir-parent\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273606    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b546ae66-9568-4840-a4e5-659d93eb80c5-run\") pod \"tuned-s6mfd\" (UID: \"b546ae66-9568-4840-a4e5-659d93eb80c5\") " pod="openshift-cluster-node-tuning-operator/tuned-s6mfd"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273669    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-registration-dir\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273676    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-host-kubelet\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273678    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8428734b-4a9f-4df9-ac06-81074198a9ba-host\") pod \"node-ca-knxtf\" (UID: \"8428734b-4a9f-4df9-ac06-81074198a9ba\") " pod="openshift-image-registry/node-ca-knxtf"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273707    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/676904dc-4d4f-4bd6-b380-357d5d9a29d4-dbus\") pod \"global-pull-secret-syncer-hwq7z\" (UID: \"676904dc-4d4f-4bd6-b380-357d5d9a29d4\") " pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273713    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-multus-socket-dir-parent\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273723    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273679    4962 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273711    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8428734b-4a9f-4df9-ac06-81074198a9ba-host\") pod \"node-ca-knxtf\" (UID: \"8428734b-4a9f-4df9-ac06-81074198a9ba\") " pod="openshift-image-registry/node-ca-knxtf"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273741    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-metrics-certs\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:20.273729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273782    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/676904dc-4d4f-4bd6-b380-357d5d9a29d4-dbus\") pod \"global-pull-secret-syncer-hwq7z\" (UID: \"676904dc-4d4f-4bd6-b380-357d5d9a29d4\") " pod="kube-system/global-pull-secret-syncer-hwq7z"
Oct 29 21:46:20.275803 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273806    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-sys-fs\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:46:20.275803 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.273824    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle podName:a2536550-3bec-4756-b872-7b14a8eb78c3 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:20.773807038 +0000 UTC m=+1.846763994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle") pod "router-default-6d4df84b7f-nm9mn" (UID: "a2536550-3bec-4756-b872-7b14a8eb78c3") : configmap references non-existent config key: service-ca.crt
Oct 29 21:46:20.275803 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273851    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc992647-89dd-44cc-ba19-3fdb03e125de-hosts-file\") pod \"node-resolver-rrmxb\" (UID: \"fc992647-89dd-44cc-ba19-3fdb03e125de\") " pod="openshift-dns/node-resolver-rrmxb"
Oct 29 21:46:20.275803 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273855    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-sys-fs\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:46:20.275803 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273885    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-run-netns\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.275803 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273899    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc992647-89dd-44cc-ba19-3fdb03e125de-hosts-file\") pod \"node-resolver-rrmxb\" (UID: \"fc992647-89dd-44cc-ba19-3fdb03e125de\") " pod="openshift-dns/node-resolver-rrmxb"
Oct 29 21:46:20.275803 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273913    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-run-multus-certs\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.275803 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273938    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-run-netns\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.275803 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273941    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-var-lib-openvswitch\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.275803 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273974    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-var-lib-openvswitch\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.275803 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273980    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-host-run-multus-certs\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.275803 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.273985    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:20.275803 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.274026    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-etc-selinux\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:46:20.275803 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.274051    4962 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 21:46:20.275803 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.274063    4962 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55df9ddfdb-9gj4j: secret "image-registry-tls" not found
Oct 29 21:46:20.275803 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.274095    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ce551e36-0249-4723-b814-a98f0a282eed-etc-selinux\") pod \"aws-ebs-csi-driver-node-bcsmz\" (UID: \"ce551e36-0249-4723-b814-a98f0a282eed\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcsmz"
Oct 29 21:46:20.275803 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.274102    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls podName:81d7056d-8cf4-4058-a8fa-1731a0b93ff6 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:20.774087533 +0000 UTC m=+1.847044491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls") pod "image-registry-55df9ddfdb-9gj4j" (UID: "81d7056d-8cf4-4058-a8fa-1731a0b93ff6") : secret "image-registry-tls" not found
Oct 29 21:46:20.275803 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.274157    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-run-ovn\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.275803 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.274052    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-run-ovn\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.275803 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.274538    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-cnibin\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.275803 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.274556    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-etc-openvswitch\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.275803 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.274597    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/259e6b95-d64c-4101-871f-4a933d4ac38f-etc-openvswitch\") pod \"ovnkube-node-74bdq\" (UID: \"259e6b95-d64c-4101-871f-4a933d4ac38f\") " pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:20.275803 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.274647    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814-cnibin\") pod \"multus-ts2kk\" (UID: \"9ae48f1b-bbf0-4e82-8eeb-33f9c1f27814\") " pod="openshift-multus/multus-ts2kk"
Oct 29 21:46:20.275803 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.275758    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0febea83-955d-4a56-8bc4-c883a4141109-metrics-certs\") pod \"network-metrics-daemon-l7tkk\" (UID: \"0febea83-955d-4a56-8bc4-c883a4141109\") " pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:46:20.276441 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.275952    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2536550-3bec-4756-b872-7b14a8eb78c3-metrics-certs\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:20.344831 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.344802    4962 scope.go:122] "RemoveContainer" containerID="b727bfa5392f31848f4d9bdd2173e4c65b562208ae7f10f517776dd67c5132f8"
Oct 29 21:46:20.355661 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.355635    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l7tkk"
Oct 29 21:46:20.696694 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.696660    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l7tkk"]
Oct 29 21:46:20.700057 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:20.700024    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0febea83_955d_4a56_8bc4_c883a4141109.slice/crio-846380b1afec8722360f2ad108193127c475c7cfeb6df0247b9edb187d4e53b8 WatchSource:0}: Error finding container 846380b1afec8722360f2ad108193127c475c7cfeb6df0247b9edb187d4e53b8: Status 404 returned error can't find the container with id 846380b1afec8722360f2ad108193127c475c7cfeb6df0247b9edb187d4e53b8
Oct 29 21:46:20.702003 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.701986    4962 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 21:46:20.779812 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.779785    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:20.779896 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.779824    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:20.779896 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.779858    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6p66l\" (UID: \"1eb2f410-512d-4e3e-838a-87c1e3b47207\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"
Oct 29 21:46:20.779896 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.779882    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9np86\" (UID: \"72352328-6f28-47ae-9190-0176cedaed4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9np86"
Oct 29 21:46:20.780026 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.779898    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:20.780026 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.779915    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:20.780026 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.779936    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle podName:a2536550-3bec-4756-b872-7b14a8eb78c3 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:21.779914284 +0000 UTC m=+2.852871228 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle") pod "router-default-6d4df84b7f-nm9mn" (UID: "a2536550-3bec-4756-b872-7b14a8eb78c3") : configmap references non-existent config key: service-ca.crt
Oct 29 21:46:20.780026 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.779979    4962 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 29 21:46:20.780026 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.779989    4962 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 21:46:20.780026 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.780004    4962 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55df9ddfdb-9gj4j: secret "image-registry-tls" not found
Oct 29 21:46:20.780026 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.780006    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d45cab6-698a-4a61-be2c-58fe7d41c55c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:20.780026 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.780028    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls podName:1eb2f410-512d-4e3e-838a-87c1e3b47207 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:21.780017171 +0000 UTC m=+2.852974112 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-6p66l" (UID: "1eb2f410-512d-4e3e-838a-87c1e3b47207") : secret "cluster-monitoring-operator-tls" not found
Oct 29 21:46:20.780419 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.780053    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls podName:81d7056d-8cf4-4058-a8fa-1731a0b93ff6 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:21.780041109 +0000 UTC m=+2.852998069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls") pod "image-registry-55df9ddfdb-9gj4j" (UID: "81d7056d-8cf4-4058-a8fa-1731a0b93ff6") : secret "image-registry-tls" not found
Oct 29 21:46:20.780419 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.780073    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nbmt9\" (UID: \"0388d722-6c0b-4cf0-9f0f-7ee369004008\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:20.780419 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.780092    4962 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 21:46:20.780419 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.780105    4962 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-547f4fc48c-tpxdx: secret "image-registry-tls" not found
Oct 29 21:46:20.780419 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.780105    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99c9881d-05b3-4953-b484-3ce51520b84a-metrics-tls\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:20.780419 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.780127    4962 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 29 21:46:20.780419 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.780138    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls podName:4b5fdb27-5000-4cf8-a025-13ae5389d17f nodeName:}" failed. No retries permitted until 2026-10-29 21:46:21.780125317 +0000 UTC m=+2.853082262 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls") pod "image-registry-547f4fc48c-tpxdx" (UID: "4b5fdb27-5000-4cf8-a025-13ae5389d17f") : secret "image-registry-tls" not found
Oct 29 21:46:20.780419 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.780187    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert podName:72352328-6f28-47ae-9190-0176cedaed4a nodeName:}" failed. No retries permitted until 2026-10-29 21:46:21.780174591 +0000 UTC m=+2.853131555 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9np86" (UID: "72352328-6f28-47ae-9190-0176cedaed4a") : secret "networking-console-plugin-cert" not found
Oct 29 21:46:20.780419 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.780195    4962 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:20.780419 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.780226    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls podName:8d45cab6-698a-4a61-be2c-58fe7d41c55c nodeName:}" failed. No retries permitted until 2026-10-29 21:46:21.780215016 +0000 UTC m=+2.853171958 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtgwl" (UID: "8d45cab6-698a-4a61-be2c-58fe7d41c55c") : secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:20.780419 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.780228    4962 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 29 21:46:20.780419 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:20.780278    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls podName:0388d722-6c0b-4cf0-9f0f-7ee369004008 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:21.780267237 +0000 UTC m=+2.853224183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-nbmt9" (UID: "0388d722-6c0b-4cf0-9f0f-7ee369004008") : secret "samples-operator-tls" not found
Oct 29 21:46:20.780820 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.780536    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8d45cab6-698a-4a61-be2c-58fe7d41c55c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:20.782179 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.782163    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/99c9881d-05b3-4953-b484-3ce51520b84a-metrics-tls\") pod \"dns-default-swlfh\" (UID: \"99c9881d-05b3-4953-b484-3ce51520b84a\") " pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:20.957490 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:20.957440    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:21.108523 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:21.108501    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/2.log"
Oct 29 21:46:21.108934 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:21.108920    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/1.log"
Oct 29 21:46:21.108995 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:21.108953    4962 generic.go:356] "Generic (PLEG): container finished" podID="f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600" containerID="d1c66b416beae5f979ca52e62c8b2aad24a0b526fc3567dca3cd6317596b36d7" exitCode=255
Oct 29 21:46:21.109062 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:21.109040    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8" event={"ID":"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600","Type":"ContainerDied","Data":"d1c66b416beae5f979ca52e62c8b2aad24a0b526fc3567dca3cd6317596b36d7"}
Oct 29 21:46:21.109123 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:21.109111    4962 scope.go:122] "RemoveContainer" containerID="b727bfa5392f31848f4d9bdd2173e4c65b562208ae7f10f517776dd67c5132f8"
Oct 29 21:46:21.109463 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:21.109447    4962 scope.go:122] "RemoveContainer" containerID="d1c66b416beae5f979ca52e62c8b2aad24a0b526fc3567dca3cd6317596b36d7"
Oct 29 21:46:21.109660 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:21.109641    4962 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jdps8_openshift-console-operator(f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8" podUID="f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600"
Oct 29 21:46:21.110278 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:21.110254    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l7tkk" event={"ID":"0febea83-955d-4a56-8bc4-c883a4141109","Type":"ContainerStarted","Data":"846380b1afec8722360f2ad108193127c475c7cfeb6df0247b9edb187d4e53b8"}
Oct 29 21:46:21.286504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:21.286476    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-swlfh"]
Oct 29 21:46:21.290140 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:21.290115    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c9881d_05b3_4953_b484_3ce51520b84a.slice/crio-2222d38318718743645f23442e0e756dd56fa6feeaccf9ac14704af4ffb94cdd WatchSource:0}: Error finding container 2222d38318718743645f23442e0e756dd56fa6feeaccf9ac14704af4ffb94cdd: Status 404 returned error can't find the container with id 2222d38318718743645f23442e0e756dd56fa6feeaccf9ac14704af4ffb94cdd
Oct 29 21:46:21.763581 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:21.762658    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:21.789029 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:21.789005    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6p66l\" (UID: \"1eb2f410-512d-4e3e-838a-87c1e3b47207\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"
Oct 29 21:46:21.789234 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:21.789211    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9np86\" (UID: \"72352328-6f28-47ae-9190-0176cedaed4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9np86"
Oct 29 21:46:21.789337 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:21.789255    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:21.789337 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:21.789314    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:21.789438 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:21.789375    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nbmt9\" (UID: \"0388d722-6c0b-4cf0-9f0f-7ee369004008\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:21.789438 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:21.789424    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:21.789547 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:21.789453    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:21.789547 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:21.789527    4962 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 21:46:21.789547 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:21.789543    4962 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-547f4fc48c-tpxdx: secret "image-registry-tls" not found
Oct 29 21:46:21.789684 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:21.789558    4962 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 21:46:21.789684 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:21.789567    4962 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55df9ddfdb-9gj4j: secret "image-registry-tls" not found
Oct 29 21:46:21.789684 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:21.789596    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls podName:4b5fdb27-5000-4cf8-a025-13ae5389d17f nodeName:}" failed. No retries permitted until 2026-10-29 21:46:23.789578449 +0000 UTC m=+4.862535397 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls") pod "image-registry-547f4fc48c-tpxdx" (UID: "4b5fdb27-5000-4cf8-a025-13ae5389d17f") : secret "image-registry-tls" not found
Oct 29 21:46:21.789684 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:21.789613    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls podName:81d7056d-8cf4-4058-a8fa-1731a0b93ff6 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:23.789603927 +0000 UTC m=+4.862560872 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls") pod "image-registry-55df9ddfdb-9gj4j" (UID: "81d7056d-8cf4-4058-a8fa-1731a0b93ff6") : secret "image-registry-tls" not found
Oct 29 21:46:21.789684 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:21.789642    4962 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:21.789929 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:21.789740    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls podName:8d45cab6-698a-4a61-be2c-58fe7d41c55c nodeName:}" failed. No retries permitted until 2026-10-29 21:46:23.789728 +0000 UTC m=+4.862684956 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtgwl" (UID: "8d45cab6-698a-4a61-be2c-58fe7d41c55c") : secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:21.789929 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:21.789801    4962 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 29 21:46:21.789929 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:21.789832    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls podName:0388d722-6c0b-4cf0-9f0f-7ee369004008 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:23.789822648 +0000 UTC m=+4.862779605 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-nbmt9" (UID: "0388d722-6c0b-4cf0-9f0f-7ee369004008") : secret "samples-operator-tls" not found
Oct 29 21:46:21.789929 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:21.789897    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle podName:a2536550-3bec-4756-b872-7b14a8eb78c3 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:23.789886303 +0000 UTC m=+4.862843262 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle") pod "router-default-6d4df84b7f-nm9mn" (UID: "a2536550-3bec-4756-b872-7b14a8eb78c3") : configmap references non-existent config key: service-ca.crt
Oct 29 21:46:21.789929 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:21.789662    4962 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 29 21:46:21.790175 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:21.789932    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert podName:72352328-6f28-47ae-9190-0176cedaed4a nodeName:}" failed. No retries permitted until 2026-10-29 21:46:23.789923406 +0000 UTC m=+4.862880363 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9np86" (UID: "72352328-6f28-47ae-9190-0176cedaed4a") : secret "networking-console-plugin-cert" not found
Oct 29 21:46:21.791995 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:21.791974    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1eb2f410-512d-4e3e-838a-87c1e3b47207-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-6p66l\" (UID: \"1eb2f410-512d-4e3e-838a-87c1e3b47207\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"
Oct 29 21:46:21.843552 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:21.843401    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"
Oct 29 21:46:22.116527 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:22.116448    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l7tkk" event={"ID":"0febea83-955d-4a56-8bc4-c883a4141109","Type":"ContainerStarted","Data":"bd56ee1e7330d1944635ff3925e70fadc9293f39f26a3f87adb7c64fa40d319c"}
Oct 29 21:46:22.116527 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:22.116517    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l7tkk" event={"ID":"0febea83-955d-4a56-8bc4-c883a4141109","Type":"ContainerStarted","Data":"b727f6037b1b944177a63b4bd4bbc06cf5a62359c2143bb7215d94c09066969d"}
Oct 29 21:46:22.117713 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:22.117681    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-swlfh" event={"ID":"99c9881d-05b3-4953-b484-3ce51520b84a","Type":"ContainerStarted","Data":"2222d38318718743645f23442e0e756dd56fa6feeaccf9ac14704af4ffb94cdd"}
Oct 29 21:46:22.119725 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:22.119696    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/2.log"
Oct 29 21:46:22.145060 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:22.145035    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l"]
Oct 29 21:46:22.150470 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:22.150442    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb2f410_512d_4e3e_838a_87c1e3b47207.slice/crio-ad5217f0e1c70d4eed9306e8164452e36f68c11d6a8d069c251802f6a7ee41e3 WatchSource:0}: Error finding container ad5217f0e1c70d4eed9306e8164452e36f68c11d6a8d069c251802f6a7ee41e3: Status 404 returned error can't find the container with id ad5217f0e1c70d4eed9306e8164452e36f68c11d6a8d069c251802f6a7ee41e3
Oct 29 21:46:23.124777 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:23.124689    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l" event={"ID":"1eb2f410-512d-4e3e-838a-87c1e3b47207","Type":"ContainerStarted","Data":"ad5217f0e1c70d4eed9306e8164452e36f68c11d6a8d069c251802f6a7ee41e3"}
Oct 29 21:46:23.126676 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:23.126650    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-swlfh" event={"ID":"99c9881d-05b3-4953-b484-3ce51520b84a","Type":"ContainerStarted","Data":"c515281cdcdc5cccd003adeec189ac8448a9db8469ac2eef2c2e58090e0d1b66"}
Oct 29 21:46:23.126819 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:23.126703    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-swlfh" event={"ID":"99c9881d-05b3-4953-b484-3ce51520b84a","Type":"ContainerStarted","Data":"5114484fe334dd24b0fc87f9bc4fa9e0b7fd84a60716018508660ec7df72f683"}
Oct 29 21:46:23.810321 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:23.810280    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nbmt9\" (UID: \"0388d722-6c0b-4cf0-9f0f-7ee369004008\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:23.810423 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:23.810355    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:23.810423 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:23.810384    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:23.810423 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:23.810390    4962 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 29 21:46:23.810610 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:23.810424    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9np86\" (UID: \"72352328-6f28-47ae-9190-0176cedaed4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9np86"
Oct 29 21:46:23.810610 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:23.810438    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls podName:0388d722-6c0b-4cf0-9f0f-7ee369004008 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:27.810423962 +0000 UTC m=+8.883380929 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-nbmt9" (UID: "0388d722-6c0b-4cf0-9f0f-7ee369004008") : secret "samples-operator-tls" not found
Oct 29 21:46:23.810610 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:23.810475    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:23.810610 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:23.810499    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:23.810610 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:23.810520    4962 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 21:46:23.810610 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:23.810532    4962 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55df9ddfdb-9gj4j: secret "image-registry-tls" not found
Oct 29 21:46:23.810610 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:23.810574    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls podName:81d7056d-8cf4-4058-a8fa-1731a0b93ff6 nodeName:}" failed. No retries permitted until 2026-10-29 21:46:27.810563458 +0000 UTC m=+8.883520417 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls") pod "image-registry-55df9ddfdb-9gj4j" (UID: "81d7056d-8cf4-4058-a8fa-1731a0b93ff6") : secret "image-registry-tls" not found
Oct 29 21:46:23.810610 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:23.810581    4962 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 29 21:46:23.810610 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:23.810600    4962 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:23.810610 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:23.810612    4962 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 29 21:46:23.810947 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:23.810627    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert podName:72352328-6f28-47ae-9190-0176cedaed4a nodeName:}" failed. No retries permitted until 2026-10-29 21:46:27.810613828 +0000 UTC m=+8.883570774 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-9np86" (UID: "72352328-6f28-47ae-9190-0176cedaed4a") : secret "networking-console-plugin-cert" not found
Oct 29 21:46:23.810947 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:23.810628    4962 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-547f4fc48c-tpxdx: secret "image-registry-tls" not found
Oct 29 21:46:23.810947 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:23.810641    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls podName:8d45cab6-698a-4a61-be2c-58fe7d41c55c nodeName:}" failed. No retries permitted until 2026-10-29 21:46:27.810635207 +0000 UTC m=+8.883592148 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtgwl" (UID: "8d45cab6-698a-4a61-be2c-58fe7d41c55c") : secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:23.810947 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:23.810677    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls podName:4b5fdb27-5000-4cf8-a025-13ae5389d17f nodeName:}" failed. No retries permitted until 2026-10-29 21:46:27.810662096 +0000 UTC m=+8.883619047 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls") pod "image-registry-547f4fc48c-tpxdx" (UID: "4b5fdb27-5000-4cf8-a025-13ae5389d17f") : secret "image-registry-tls" not found
Oct 29 21:46:23.811075 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:23.810964    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2536550-3bec-4756-b872-7b14a8eb78c3-service-ca-bundle\") pod \"router-default-6d4df84b7f-nm9mn\" (UID: \"a2536550-3bec-4756-b872-7b14a8eb78c3\") " pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:23.813774 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:23.813758    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-g2rdv"
Oct 29 21:46:23.946218 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:23.946190    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:23.998971 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:23.998813    4962 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2rdv_af572bfb-940b-4226-b91f-d06554b40d3c/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:24.127166 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:24.127091    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:24.127609 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:24.127522    4962 scope.go:122] "RemoveContainer" containerID="d1c66b416beae5f979ca52e62c8b2aad24a0b526fc3567dca3cd6317596b36d7"
Oct 29 21:46:24.127738 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:24.127713    4962 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jdps8_openshift-console-operator(f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8" podUID="f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600"
Oct 29 21:46:24.131513 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:24.131481    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-6p66l" event={"ID":"1eb2f410-512d-4e3e-838a-87c1e3b47207","Type":"ContainerStarted","Data":"2e1d92990d96440d441be0cdc49880c8af748ba72df8cb210fcafc2bdfb4f936"}
Oct 29 21:46:24.133014 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:24.132992    4962 generic.go:356] "Generic (PLEG): container finished" podID="0e7f3fff-da9c-4189-bd53-5d6586ee0a4c" containerID="c246dd289b948d0e2ab7032057d4957492710b93e75b7cac748d124d08a45c2a" exitCode=0
Oct 29 21:46:24.133108 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:24.133049    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k" event={"ID":"0e7f3fff-da9c-4189-bd53-5d6586ee0a4c","Type":"ContainerDied","Data":"c246dd289b948d0e2ab7032057d4957492710b93e75b7cac748d124d08a45c2a"}
Oct 29 21:46:24.133194 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:24.133183    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:24.133524 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:24.133509    4962 scope.go:122] "RemoveContainer" containerID="c246dd289b948d0e2ab7032057d4957492710b93e75b7cac748d124d08a45c2a"
Oct 29 21:46:24.225505 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:24.225473    4962 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2rdv_af572bfb-940b-4226-b91f-d06554b40d3c/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:24.261570 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:24.261546    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6d4df84b7f-nm9mn"]
Oct 29 21:46:24.264548 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:24.264524    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2536550_3bec_4756_b872_7b14a8eb78c3.slice/crio-142995881c365e6417d285768e4334b458599e310c141d79c13b5be0528fa5a8 WatchSource:0}: Error finding container 142995881c365e6417d285768e4334b458599e310c141d79c13b5be0528fa5a8: Status 404 returned error can't find the container with id 142995881c365e6417d285768e4334b458599e310c141d79c13b5be0528fa5a8
Oct 29 21:46:24.419142 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:24.419111    4962 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2rdv_af572bfb-940b-4226-b91f-d06554b40d3c/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:24.579462 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:24.579392    4962 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2rdv_af572bfb-940b-4226-b91f-d06554b40d3c/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:24.810998 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:24.810925    4962 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-g2rdv_af572bfb-940b-4226-b91f-d06554b40d3c/konnectivity-agent/0.log" limit=1024
Oct 29 21:46:25.137566 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:25.137486    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6d4df84b7f-nm9mn" event={"ID":"a2536550-3bec-4756-b872-7b14a8eb78c3","Type":"ContainerStarted","Data":"9738e45514a7aeda06e58a932485f0b6811c29d7167938c6b4ad3dae17cbe62e"}
Oct 29 21:46:25.138017 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:25.137541    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6d4df84b7f-nm9mn" event={"ID":"a2536550-3bec-4756-b872-7b14a8eb78c3","Type":"ContainerStarted","Data":"142995881c365e6417d285768e4334b458599e310c141d79c13b5be0528fa5a8"}
Oct 29 21:46:25.139253 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:25.139227    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-d7x4k" event={"ID":"0e7f3fff-da9c-4189-bd53-5d6586ee0a4c","Type":"ContainerStarted","Data":"bf045bf9c9018cc445a0ad16f533528cde6f75ebb3b0e0411ad6d60f72b9038a"}
Oct 29 21:46:25.947171 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:25.947139    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:25.949408 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:25.949388    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:27.144975 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:27.144946    4962 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 21:46:27.289469 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:27.289445    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:27.289683 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:27.289668    4962 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 21:46:27.321191 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:27.321164    4962 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:27.321481 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:27.321468    4962 scope.go:122] "RemoveContainer" containerID="d1c66b416beae5f979ca52e62c8b2aad24a0b526fc3567dca3cd6317596b36d7"
Oct 29 21:46:27.321625 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:27.321608    4962 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-jdps8_openshift-console-operator(f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8" podUID="f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600"
Oct 29 21:46:27.843970 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:27.843936    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:27.844168 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:27.843986    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9np86\" (UID: \"72352328-6f28-47ae-9190-0176cedaed4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9np86"
Oct 29 21:46:27.844168 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:27.844097    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:27.844168 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:27.844138    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:27.844362 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:27.844192    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nbmt9\" (UID: \"0388d722-6c0b-4cf0-9f0f-7ee369004008\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:27.844362 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:27.844309    4962 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:27.844466 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:27.844386    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls podName:8d45cab6-698a-4a61-be2c-58fe7d41c55c nodeName:}" failed. No retries permitted until 2026-10-29 21:46:35.844370452 +0000 UTC m=+16.917327412 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dtgwl" (UID: "8d45cab6-698a-4a61-be2c-58fe7d41c55c") : secret "insights-runtime-extractor-tls" not found
Oct 29 21:46:27.846595 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:27.846570    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0388d722-6c0b-4cf0-9f0f-7ee369004008-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-nbmt9\" (UID: \"0388d722-6c0b-4cf0-9f0f-7ee369004008\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:27.846706 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:27.846570    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72352328-6f28-47ae-9190-0176cedaed4a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-9np86\" (UID: \"72352328-6f28-47ae-9190-0176cedaed4a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-9np86"
Oct 29 21:46:27.846969 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:27.846949    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls\") pod \"image-registry-547f4fc48c-tpxdx\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") " pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:27.847002 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:27.846949    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls\") pod \"image-registry-55df9ddfdb-9gj4j\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") " pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:27.856726 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:27.856706    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:28.143546 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:28.143479    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"
Oct 29 21:46:28.146374 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:28.146350    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:28.146778 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:28.146685    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-9np86"
Oct 29 21:46:28.182758 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:28.182678    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55df9ddfdb-9gj4j"]
Oct 29 21:46:28.203729 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:28.202786    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d7056d_8cf4_4058_a8fa_1731a0b93ff6.slice/crio-665486255dfe48b35c60664417004e9dba8cd997383a3da17b0485c3ebb07e59 WatchSource:0}: Error finding container 665486255dfe48b35c60664417004e9dba8cd997383a3da17b0485c3ebb07e59: Status 404 returned error can't find the container with id 665486255dfe48b35c60664417004e9dba8cd997383a3da17b0485c3ebb07e59
Oct 29 21:46:28.261835 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:28.261817    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:28.262004 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:28.261931    4962 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 21:46:28.263139 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:28.263114    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6d4df84b7f-nm9mn"
Oct 29 21:46:28.456393 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:28.456367    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9"]
Oct 29 21:46:28.459567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:28.459540    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-547f4fc48c-tpxdx"]
Oct 29 21:46:28.463571 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:28.463388    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b5fdb27_5000_4cf8_a025_13ae5389d17f.slice/crio-da85f30c6e6f72b0cc9902506d4619a63cfb4018fe638554f1c539576f010279 WatchSource:0}: Error finding container da85f30c6e6f72b0cc9902506d4619a63cfb4018fe638554f1c539576f010279: Status 404 returned error can't find the container with id da85f30c6e6f72b0cc9902506d4619a63cfb4018fe638554f1c539576f010279
Oct 29 21:46:28.464936 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:28.464907    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-9np86"]
Oct 29 21:46:28.472631 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:28.472608    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72352328_6f28_47ae_9190_0176cedaed4a.slice/crio-4bb6424399c795f3f909fe2570d4cb7dd6a69821a50806805b7d7163d50f8214 WatchSource:0}: Error finding container 4bb6424399c795f3f909fe2570d4cb7dd6a69821a50806805b7d7163d50f8214: Status 404 returned error can't find the container with id 4bb6424399c795f3f909fe2570d4cb7dd6a69821a50806805b7d7163d50f8214
Oct 29 21:46:29.155330 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:29.155270    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9" event={"ID":"0388d722-6c0b-4cf0-9f0f-7ee369004008","Type":"ContainerStarted","Data":"1814419c0bf2893c0596a320d16fd1bc4c88909b8d2f4eb734b4f7c7f645a418"}
Oct 29 21:46:29.158206 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:29.158173    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j" event={"ID":"81d7056d-8cf4-4058-a8fa-1731a0b93ff6","Type":"ContainerStarted","Data":"4eeabd27b18f4b358ce07da09495036d184c136581e3837a802d98c897c55b29"}
Oct 29 21:46:29.158360 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:29.158225    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j" event={"ID":"81d7056d-8cf4-4058-a8fa-1731a0b93ff6","Type":"ContainerStarted","Data":"665486255dfe48b35c60664417004e9dba8cd997383a3da17b0485c3ebb07e59"}
Oct 29 21:46:29.159561 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:29.159538    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-9np86" event={"ID":"72352328-6f28-47ae-9190-0176cedaed4a","Type":"ContainerStarted","Data":"4bb6424399c795f3f909fe2570d4cb7dd6a69821a50806805b7d7163d50f8214"}
Oct 29 21:46:29.160929 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:29.160903    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx" event={"ID":"4b5fdb27-5000-4cf8-a025-13ae5389d17f","Type":"ContainerStarted","Data":"30567fabb7551812a51da21ff4a0714a7bc73bac20b7573fbff93fdad7cab28c"}
Oct 29 21:46:29.161029 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:29.160945    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx" event={"ID":"4b5fdb27-5000-4cf8-a025-13ae5389d17f","Type":"ContainerStarted","Data":"da85f30c6e6f72b0cc9902506d4619a63cfb4018fe638554f1c539576f010279"}
Oct 29 21:46:30.166031 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:30.166000    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-9np86" event={"ID":"72352328-6f28-47ae-9190-0176cedaed4a","Type":"ContainerStarted","Data":"e67cbd9491822f661d8ec094a3c1b783113e41e24a3b5f119a44f1445155092d"}
Oct 29 21:46:30.167405 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:30.167385    4962 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 21:46:30.167512 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:30.167405    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9" event={"ID":"0388d722-6c0b-4cf0-9f0f-7ee369004008","Type":"ContainerStarted","Data":"e9dc72988f3339851770693329774e7d6c6175a39e7568c1363df91910caf4fc"}
Oct 29 21:46:30.167512 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:30.167451    4962 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 21:46:30.202439 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:30.202407    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:46:31.175937 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:31.175901    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-nbmt9" event={"ID":"0388d722-6c0b-4cf0-9f0f-7ee369004008","Type":"ContainerStarted","Data":"a52857053ebc6e2e8b59fd15365838488864bec49ca240027184ca18c85ec4c8"}
Oct 29 21:46:32.051154 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:32.051127    4962 scope.go:122] "RemoveContainer" containerID="d1c66b416beae5f979ca52e62c8b2aad24a0b526fc3567dca3cd6317596b36d7"
Oct 29 21:46:32.181112 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:32.181088    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/2.log"
Oct 29 21:46:32.181522 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:32.181202    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8" event={"ID":"f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600","Type":"ContainerStarted","Data":"d2cd20603675c05ec925e8e0658ac963de8ff2a96f79d93ff0b8034771ae2ecf"}
Oct 29 21:46:32.181675 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:32.181655    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:32.182491 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:32.182468    4962 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-jdps8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused" start-of-body=
Oct 29 21:46:32.182583 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:32.182506    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8" podUID="f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.17:8443/readyz\": dial tcp 10.132.0.17:8443: connect: connection refused"
Oct 29 21:46:33.189438 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:33.189406    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8"
Oct 29 21:46:34.142222 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:34.142192    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-swlfh"
Oct 29 21:46:35.061228 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:35.061191    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:35.908837 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:35.908804    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:35.910987 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:35.910959    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8d45cab6-698a-4a61-be2c-58fe7d41c55c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dtgwl\" (UID: \"8d45cab6-698a-4a61-be2c-58fe7d41c55c\") " pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:35.958604 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:35.958580    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dtgwl"
Oct 29 21:46:35.969466 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:35.969440    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:36.257082 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:36.257002    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dtgwl"]
Oct 29 21:46:36.260011 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:36.259985    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d45cab6_698a_4a61_be2c_58fe7d41c55c.slice/crio-11425ed7d557be1cf3d16eecc6d508b86ec74b7c97bc6678814e76e6d7f88ec2 WatchSource:0}: Error finding container 11425ed7d557be1cf3d16eecc6d508b86ec74b7c97bc6678814e76e6d7f88ec2: Status 404 returned error can't find the container with id 11425ed7d557be1cf3d16eecc6d508b86ec74b7c97bc6678814e76e6d7f88ec2
Oct 29 21:46:36.328097 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:36.328073    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-74bdq"
Oct 29 21:46:37.135204 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:37.135176    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-p49l7-nlq2m"]
Oct 29 21:46:37.138652 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:37.138629    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-p49l7-nlq2m"
Oct 29 21:46:37.182753 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:37.182725    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-p49l7-nlq2m"]
Oct 29 21:46:37.202034 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:37.202005    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dtgwl" event={"ID":"8d45cab6-698a-4a61-be2c-58fe7d41c55c","Type":"ContainerStarted","Data":"a2c177449b2c1f7fc70c85d59e2d0a1085195f6d34a0f014b408daf4aa3ccfac"}
Oct 29 21:46:37.202172 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:37.202054    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dtgwl" event={"ID":"8d45cab6-698a-4a61-be2c-58fe7d41c55c","Type":"ContainerStarted","Data":"45c377bfd3d7e7085989e3decfe66fcb2afeaf62305d5ee1cb4f355b3fdd50e5"}
Oct 29 21:46:37.202172 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:37.202074    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dtgwl" event={"ID":"8d45cab6-698a-4a61-be2c-58fe7d41c55c","Type":"ContainerStarted","Data":"11425ed7d557be1cf3d16eecc6d508b86ec74b7c97bc6678814e76e6d7f88ec2"}
Oct 29 21:46:37.204944 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:37.204917    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-547f4fc48c-tpxdx"]
Oct 29 21:46:37.218773 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:37.218741    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdt58\" (UniqueName: \"kubernetes.io/projected/0b0306f7-0104-4219-9fb4-a601014c7a43-kube-api-access-vdt58\") pod \"periodic-gathering-p49l7-nlq2m\" (UID: \"0b0306f7-0104-4219-9fb4-a601014c7a43\") " pod="openshift-insights/periodic-gathering-p49l7-nlq2m"
Oct 29 21:46:37.218916 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:37.218782    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0306f7-0104-4219-9fb4-a601014c7a43-trusted-ca-bundle\") pod \"periodic-gathering-p49l7-nlq2m\" (UID: \"0b0306f7-0104-4219-9fb4-a601014c7a43\") " pod="openshift-insights/periodic-gathering-p49l7-nlq2m"
Oct 29 21:46:37.218916 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:37.218823    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0306f7-0104-4219-9fb4-a601014c7a43-service-ca-bundle\") pod \"periodic-gathering-p49l7-nlq2m\" (UID: \"0b0306f7-0104-4219-9fb4-a601014c7a43\") " pod="openshift-insights/periodic-gathering-p49l7-nlq2m"
Oct 29 21:46:37.218916 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:37.218888    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0b0306f7-0104-4219-9fb4-a601014c7a43-archives-path\") pod \"periodic-gathering-p49l7-nlq2m\" (UID: \"0b0306f7-0104-4219-9fb4-a601014c7a43\") " pod="openshift-insights/periodic-gathering-p49l7-nlq2m"
Oct 29 21:46:37.319875 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:37.319845    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0b0306f7-0104-4219-9fb4-a601014c7a43-archives-path\") pod \"periodic-gathering-p49l7-nlq2m\" (UID: \"0b0306f7-0104-4219-9fb4-a601014c7a43\") " pod="openshift-insights/periodic-gathering-p49l7-nlq2m"
Oct 29 21:46:37.320352 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:37.319927    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdt58\" (UniqueName: \"kubernetes.io/projected/0b0306f7-0104-4219-9fb4-a601014c7a43-kube-api-access-vdt58\") pod \"periodic-gathering-p49l7-nlq2m\" (UID: \"0b0306f7-0104-4219-9fb4-a601014c7a43\") " pod="openshift-insights/periodic-gathering-p49l7-nlq2m"
Oct 29 21:46:37.320352 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:37.319963    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0306f7-0104-4219-9fb4-a601014c7a43-trusted-ca-bundle\") pod \"periodic-gathering-p49l7-nlq2m\" (UID: \"0b0306f7-0104-4219-9fb4-a601014c7a43\") " pod="openshift-insights/periodic-gathering-p49l7-nlq2m"
Oct 29 21:46:37.320352 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:37.320010    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0306f7-0104-4219-9fb4-a601014c7a43-service-ca-bundle\") pod \"periodic-gathering-p49l7-nlq2m\" (UID: \"0b0306f7-0104-4219-9fb4-a601014c7a43\") " pod="openshift-insights/periodic-gathering-p49l7-nlq2m"
Oct 29 21:46:37.320352 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:37.320262    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0b0306f7-0104-4219-9fb4-a601014c7a43-archives-path\") pod \"periodic-gathering-p49l7-nlq2m\" (UID: \"0b0306f7-0104-4219-9fb4-a601014c7a43\") " pod="openshift-insights/periodic-gathering-p49l7-nlq2m"
Oct 29 21:46:37.320834 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:37.320809    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0306f7-0104-4219-9fb4-a601014c7a43-service-ca-bundle\") pod \"periodic-gathering-p49l7-nlq2m\" (UID: \"0b0306f7-0104-4219-9fb4-a601014c7a43\") " pod="openshift-insights/periodic-gathering-p49l7-nlq2m"
Oct 29 21:46:37.320834 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:37.320823    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0306f7-0104-4219-9fb4-a601014c7a43-trusted-ca-bundle\") pod \"periodic-gathering-p49l7-nlq2m\" (UID: \"0b0306f7-0104-4219-9fb4-a601014c7a43\") " pod="openshift-insights/periodic-gathering-p49l7-nlq2m"
Oct 29 21:46:37.344070 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:37.344048    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdt58\" (UniqueName: \"kubernetes.io/projected/0b0306f7-0104-4219-9fb4-a601014c7a43-kube-api-access-vdt58\") pod \"periodic-gathering-p49l7-nlq2m\" (UID: \"0b0306f7-0104-4219-9fb4-a601014c7a43\") " pod="openshift-insights/periodic-gathering-p49l7-nlq2m"
Oct 29 21:46:37.452212 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:37.452187    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-p49l7-nlq2m"
Oct 29 21:46:37.756380 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:37.756307    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-p49l7-nlq2m"]
Oct 29 21:46:38.207230 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:38.207201    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-p49l7-nlq2m" event={"ID":"0b0306f7-0104-4219-9fb4-a601014c7a43","Type":"ContainerStarted","Data":"35eb6d57641e98d1a75a7ce960c86ce2abd400b77dea64728c3443ab6bb9d034"}
Oct 29 21:46:38.207343 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:38.207251    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-p49l7-nlq2m" event={"ID":"0b0306f7-0104-4219-9fb4-a601014c7a43","Type":"ContainerStarted","Data":"2983dd1bcd7f3d015ff75d741a3741dbd5e61b6e4dab6a29d136e51acdac3325"}
Oct 29 21:46:38.249107 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:38.249048    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-p49l7-nlq2m" podStartSLOduration=1.249030554 podStartE2EDuration="1.249030554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:38.247854519 +0000 UTC m=+19.320811484" watchObservedRunningTime="2026-10-29 21:46:38.249030554 +0000 UTC m=+19.321987516"
Oct 29 21:46:38.867857 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:38.867822    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-swlfh_99c9881d-05b3-4953-b484-3ce51520b84a/dns/0.log"
Oct 29 21:46:39.073571 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:39.073496    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-swlfh_99c9881d-05b3-4953-b484-3ce51520b84a/kube-rbac-proxy/0.log"
Oct 29 21:46:39.216365 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:39.216326    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dtgwl" event={"ID":"8d45cab6-698a-4a61-be2c-58fe7d41c55c","Type":"ContainerStarted","Data":"ea11be49b7206a625851d1c846379d1edc101474f1693fe211e03b05454a5741"}
Oct 29 21:46:39.718277 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:46:39.718240    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:46:40.076202 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:40.076134    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rrmxb_fc992647-89dd-44cc-ba19-3fdb03e125de/dns-node-resolver/0.log"
Oct 29 21:46:40.268638 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:40.268608    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-547f4fc48c-tpxdx_4b5fdb27-5000-4cf8-a025-13ae5389d17f/registry/0.log"
Oct 29 21:46:40.474410 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:40.474382    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-55df9ddfdb-9gj4j_81d7056d-8cf4-4058-a8fa-1731a0b93ff6/registry/0.log"
Oct 29 21:46:41.067811 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:41.067785    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-knxtf_8428734b-4a9f-4df9-ac06-81074198a9ba/node-ca/0.log"
Oct 29 21:46:41.469977 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:41.469953    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d4df84b7f-nm9mn_a2536550-3bec-4756-b872-7b14a8eb78c3/router/0.log"
Oct 29 21:46:41.672759 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:41.672730    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dkzbq_07a537c1-669a-4a02-bc1f-f09ea24e47ca/serve-healthcheck-canary/0.log"
Oct 29 21:46:44.941830 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:44.941797    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tc8tb"]
Oct 29 21:46:44.983798 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:44.983771    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.058876 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.058848    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 29 21:46:45.059170 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.059152    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 29 21:46:45.059253 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.059170    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pjw5g\""
Oct 29 21:46:45.059253 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.059207    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 29 21:46:45.059379 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.059317    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 29 21:46:45.065398 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.065382    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:46:45.078130 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.078112    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d526d422-6b0e-46ab-a904-a4c49ae5de79-node-exporter-tls\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.078226 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.078137    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d526d422-6b0e-46ab-a904-a4c49ae5de79-metrics-client-ca\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.078226 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.078159    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr7d5\" (UniqueName: \"kubernetes.io/projected/d526d422-6b0e-46ab-a904-a4c49ae5de79-kube-api-access-fr7d5\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.078372 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.078352    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d526d422-6b0e-46ab-a904-a4c49ae5de79-node-exporter-wtmp\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.078513 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.078493    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d526d422-6b0e-46ab-a904-a4c49ae5de79-node-exporter-textfile\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.078623 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.078571    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d526d422-6b0e-46ab-a904-a4c49ae5de79-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.078692 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.078671    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d526d422-6b0e-46ab-a904-a4c49ae5de79-node-exporter-accelerators-collector-config\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.078739 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.078716    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d526d422-6b0e-46ab-a904-a4c49ae5de79-sys\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.078784 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.078768    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d526d422-6b0e-46ab-a904-a4c49ae5de79-root\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.179265 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.179234    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr7d5\" (UniqueName: \"kubernetes.io/projected/d526d422-6b0e-46ab-a904-a4c49ae5de79-kube-api-access-fr7d5\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.179414 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.179319    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d526d422-6b0e-46ab-a904-a4c49ae5de79-node-exporter-wtmp\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.179414 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.179358    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d526d422-6b0e-46ab-a904-a4c49ae5de79-node-exporter-textfile\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.179414 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.179388    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d526d422-6b0e-46ab-a904-a4c49ae5de79-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.179551 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.179415    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d526d422-6b0e-46ab-a904-a4c49ae5de79-node-exporter-accelerators-collector-config\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.179551 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.179442    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d526d422-6b0e-46ab-a904-a4c49ae5de79-sys\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.179551 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.179468    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d526d422-6b0e-46ab-a904-a4c49ae5de79-root\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.179551 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.179505    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d526d422-6b0e-46ab-a904-a4c49ae5de79-node-exporter-tls\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.179551 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.179510    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d526d422-6b0e-46ab-a904-a4c49ae5de79-node-exporter-wtmp\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.179551 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.179527    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d526d422-6b0e-46ab-a904-a4c49ae5de79-metrics-client-ca\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.179833 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.179537    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d526d422-6b0e-46ab-a904-a4c49ae5de79-sys\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.179833 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.179552    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d526d422-6b0e-46ab-a904-a4c49ae5de79-root\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.179833 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.179701    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d526d422-6b0e-46ab-a904-a4c49ae5de79-node-exporter-textfile\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.180109 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.180090    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d526d422-6b0e-46ab-a904-a4c49ae5de79-node-exporter-accelerators-collector-config\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.180188 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.180113    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d526d422-6b0e-46ab-a904-a4c49ae5de79-metrics-client-ca\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.181765 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.181745    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d526d422-6b0e-46ab-a904-a4c49ae5de79-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.181893 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.181830    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d526d422-6b0e-46ab-a904-a4c49ae5de79-node-exporter-tls\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.202173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.202126    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr7d5\" (UniqueName: \"kubernetes.io/projected/d526d422-6b0e-46ab-a904-a4c49ae5de79-kube-api-access-fr7d5\") pod \"node-exporter-tc8tb\" (UID: \"d526d422-6b0e-46ab-a904-a4c49ae5de79\") " pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.332613 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:45.332582    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tc8tb"
Oct 29 21:46:45.379063 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:46:45.379043    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd526d422_6b0e_46ab_a904_a4c49ae5de79.slice/crio-fa45df6087cf3d9980b4638ac494d107c7253e0acf271304f4e27c0cb5969e26 WatchSource:0}: Error finding container fa45df6087cf3d9980b4638ac494d107c7253e0acf271304f4e27c0cb5969e26: Status 404 returned error can't find the container with id fa45df6087cf3d9980b4638ac494d107c7253e0acf271304f4e27c0cb5969e26
Oct 29 21:46:46.244885 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:46.244835    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tc8tb" event={"ID":"d526d422-6b0e-46ab-a904-a4c49ae5de79","Type":"ContainerStarted","Data":"fa45df6087cf3d9980b4638ac494d107c7253e0acf271304f4e27c0cb5969e26"}
Oct 29 21:46:47.209408 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:47.209379    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:46:47.250667 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:47.250635    4962 generic.go:356] "Generic (PLEG): container finished" podID="d526d422-6b0e-46ab-a904-a4c49ae5de79" containerID="957aac52101849061faf29ab216758edd821941e5cdd7e057e7c67797828cd5e" exitCode=0
Oct 29 21:46:47.251115 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:47.250680    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tc8tb" event={"ID":"d526d422-6b0e-46ab-a904-a4c49ae5de79","Type":"ContainerDied","Data":"957aac52101849061faf29ab216758edd821941e5cdd7e057e7c67797828cd5e"}
Oct 29 21:46:48.257075 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:48.257038    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tc8tb" event={"ID":"d526d422-6b0e-46ab-a904-a4c49ae5de79","Type":"ContainerStarted","Data":"fa7cf2c7089bccb9042d3aa30d2d4c857a09c7964796b4a409250c5a05f4a590"}
Oct 29 21:46:48.257566 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:48.257099    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tc8tb" event={"ID":"d526d422-6b0e-46ab-a904-a4c49ae5de79","Type":"ContainerStarted","Data":"c7bc80bbb186bba1b87c7a2c57cd2a0aca4fdb3e6f795baeae73f5421a453cf9"}
Oct 29 21:46:50.205213 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:50.205187    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pkmzk"
Oct 29 21:46:50.245778 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:50.245730    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tc8tb" podStartSLOduration=4.837680709 podStartE2EDuration="6.245717993s" totalImagesPullingTime="1.321187406s" totalInitContainerRuntime="86.849878ms" isStatefulPod=true podCreationTimestamp="2026-10-29 21:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:46:48.307724634 +0000 UTC m=+29.380681598" watchObservedRunningTime="2026-10-29 21:46:50.245717993 +0000 UTC m=+31.318674950"
Oct 29 21:46:59.221137 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:46:59.221103    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55df9ddfdb-9gj4j"]
Oct 29 21:47:02.224447 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.224351    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx" podUID="4b5fdb27-5000-4cf8-a025-13ae5389d17f" containerName="registry" containerID="cri-o://30567fabb7551812a51da21ff4a0714a7bc73bac20b7573fbff93fdad7cab28c" gracePeriod=30
Oct 29 21:47:02.468423 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.468399    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:47:02.514938 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.514881    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b5fdb27-5000-4cf8-a025-13ae5389d17f-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4b5fdb27-5000-4cf8-a025-13ae5389d17f-image-registry-private-configuration\") pod \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") "
Oct 29 21:47:02.514938 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.514915    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b5fdb27-5000-4cf8-a025-13ae5389d17f-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4b5fdb27-5000-4cf8-a025-13ae5389d17f-ca-trust-extracted\") pod \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") "
Oct 29 21:47:02.515092 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.514945    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b5fdb27-5000-4cf8-a025-13ae5389d17f-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b5fdb27-5000-4cf8-a025-13ae5389d17f-trusted-ca\") pod \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") "
Oct 29 21:47:02.515092 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.515001    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-kube-api-access-69xwz\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-kube-api-access-69xwz\") pod \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") "
Oct 29 21:47:02.515092 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.515025    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b5fdb27-5000-4cf8-a025-13ae5389d17f-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4b5fdb27-5000-4cf8-a025-13ae5389d17f-installation-pull-secrets\") pod \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") "
Oct 29 21:47:02.515244 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.515089    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls\") pod \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") "
Oct 29 21:47:02.515244 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.515149    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-certificates\") pod \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") "
Oct 29 21:47:02.515375 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.515184    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-bound-sa-token\") pod \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\" (UID: \"4b5fdb27-5000-4cf8-a025-13ae5389d17f\") "
Oct 29 21:47:02.516054 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.515576    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b5fdb27-5000-4cf8-a025-13ae5389d17f-trusted-ca" pod "4b5fdb27-5000-4cf8-a025-13ae5389d17f" (UID: "4b5fdb27-5000-4cf8-a025-13ae5389d17f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:47:02.516054 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.516007    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-certificates" pod "4b5fdb27-5000-4cf8-a025-13ae5389d17f" (UID: "4b5fdb27-5000-4cf8-a025-13ae5389d17f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:47:02.517777 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.517728    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b5fdb27-5000-4cf8-a025-13ae5389d17f-installation-pull-secrets" pod "4b5fdb27-5000-4cf8-a025-13ae5389d17f" (UID: "4b5fdb27-5000-4cf8-a025-13ae5389d17f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:47:02.517892 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.517781    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-kube-api-access-69xwz" pod "4b5fdb27-5000-4cf8-a025-13ae5389d17f" (UID: "4b5fdb27-5000-4cf8-a025-13ae5389d17f"). InnerVolumeSpecName "kube-api-access-69xwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:47:02.517892 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.517810    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b5fdb27-5000-4cf8-a025-13ae5389d17f-image-registry-private-configuration" pod "4b5fdb27-5000-4cf8-a025-13ae5389d17f" (UID: "4b5fdb27-5000-4cf8-a025-13ae5389d17f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:47:02.518091 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.518075    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls" pod "4b5fdb27-5000-4cf8-a025-13ae5389d17f" (UID: "4b5fdb27-5000-4cf8-a025-13ae5389d17f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:47:02.518331 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.518307    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-bound-sa-token" pod "4b5fdb27-5000-4cf8-a025-13ae5389d17f" (UID: "4b5fdb27-5000-4cf8-a025-13ae5389d17f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:47:02.522175 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.522161    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b5fdb27-5000-4cf8-a025-13ae5389d17f-ca-trust-extracted" pod "4b5fdb27-5000-4cf8-a025-13ae5389d17f" (UID: "4b5fdb27-5000-4cf8-a025-13ae5389d17f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:47:02.616927 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.616883    4962 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:47:02.616927 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.616927    4962 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4b5fdb27-5000-4cf8-a025-13ae5389d17f-registry-certificates\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:47:02.616927 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.616939    4962 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-bound-sa-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:47:02.617123 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.616949    4962 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4b5fdb27-5000-4cf8-a025-13ae5389d17f-image-registry-private-configuration\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:47:02.617123 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.616958    4962 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4b5fdb27-5000-4cf8-a025-13ae5389d17f-ca-trust-extracted\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:47:02.617123 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.616967    4962 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b5fdb27-5000-4cf8-a025-13ae5389d17f-trusted-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:47:02.617123 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.616975    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69xwz\" (UniqueName: \"kubernetes.io/projected/4b5fdb27-5000-4cf8-a025-13ae5389d17f-kube-api-access-69xwz\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:47:02.617123 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:02.616983    4962 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4b5fdb27-5000-4cf8-a025-13ae5389d17f-installation-pull-secrets\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:47:03.325253 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:03.325179    4962 generic.go:356] "Generic (PLEG): container finished" podID="4b5fdb27-5000-4cf8-a025-13ae5389d17f" containerID="30567fabb7551812a51da21ff4a0714a7bc73bac20b7573fbff93fdad7cab28c" exitCode=0
Oct 29 21:47:03.325253 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:03.325233    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx"
Oct 29 21:47:03.325724 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:03.325255    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx" event={"ID":"4b5fdb27-5000-4cf8-a025-13ae5389d17f","Type":"ContainerDied","Data":"30567fabb7551812a51da21ff4a0714a7bc73bac20b7573fbff93fdad7cab28c"}
Oct 29 21:47:03.325724 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:03.325318    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-547f4fc48c-tpxdx" event={"ID":"4b5fdb27-5000-4cf8-a025-13ae5389d17f","Type":"ContainerDied","Data":"da85f30c6e6f72b0cc9902506d4619a63cfb4018fe638554f1c539576f010279"}
Oct 29 21:47:03.325724 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:03.325342    4962 scope.go:122] "RemoveContainer" containerID="30567fabb7551812a51da21ff4a0714a7bc73bac20b7573fbff93fdad7cab28c"
Oct 29 21:47:03.334897 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:03.334880    4962 scope.go:122] "RemoveContainer" containerID="30567fabb7551812a51da21ff4a0714a7bc73bac20b7573fbff93fdad7cab28c"
Oct 29 21:47:03.335137 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:47:03.335120    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30567fabb7551812a51da21ff4a0714a7bc73bac20b7573fbff93fdad7cab28c\": container with ID starting with 30567fabb7551812a51da21ff4a0714a7bc73bac20b7573fbff93fdad7cab28c not found: ID does not exist" containerID="30567fabb7551812a51da21ff4a0714a7bc73bac20b7573fbff93fdad7cab28c"
Oct 29 21:47:03.335201 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:03.335148    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30567fabb7551812a51da21ff4a0714a7bc73bac20b7573fbff93fdad7cab28c"} err="failed to get container status \"30567fabb7551812a51da21ff4a0714a7bc73bac20b7573fbff93fdad7cab28c\": rpc error: code = NotFound desc = could not find container \"30567fabb7551812a51da21ff4a0714a7bc73bac20b7573fbff93fdad7cab28c\": container with ID starting with 30567fabb7551812a51da21ff4a0714a7bc73bac20b7573fbff93fdad7cab28c not found: ID does not exist"
Oct 29 21:47:03.387749 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:03.387724    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-547f4fc48c-tpxdx"]
Oct 29 21:47:03.399520 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:03.399497    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-547f4fc48c-tpxdx"]
Oct 29 21:47:05.055511 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:05.055477    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b5fdb27-5000-4cf8-a025-13ae5389d17f" path="/var/lib/kubelet/pods/4b5fdb27-5000-4cf8-a025-13ae5389d17f/volumes"
Oct 29 21:47:09.718850 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:47:09.718814    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:47:24.244788 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.244750    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j" podUID="81d7056d-8cf4-4058-a8fa-1731a0b93ff6" containerName="registry" containerID="cri-o://4eeabd27b18f4b358ce07da09495036d184c136581e3837a802d98c897c55b29" gracePeriod=30
Oct 29 21:47:24.412818 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.412783    4962 generic.go:356] "Generic (PLEG): container finished" podID="81d7056d-8cf4-4058-a8fa-1731a0b93ff6" containerID="4eeabd27b18f4b358ce07da09495036d184c136581e3837a802d98c897c55b29" exitCode=0
Oct 29 21:47:24.412970 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.412843    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j" event={"ID":"81d7056d-8cf4-4058-a8fa-1731a0b93ff6","Type":"ContainerDied","Data":"4eeabd27b18f4b358ce07da09495036d184c136581e3837a802d98c897c55b29"}
Oct 29 21:47:24.479737 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.479711    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:47:24.562989 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.562936    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-kube-api-access-xntk2\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-kube-api-access-xntk2\") pod \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") "
Oct 29 21:47:24.562989 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.562975    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-installation-pull-secrets\") pod \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") "
Oct 29 21:47:24.563140 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.563012    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-certificates\") pod \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") "
Oct 29 21:47:24.563140 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.563036    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls\") pod \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") "
Oct 29 21:47:24.563140 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.563068    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-ca-trust-extracted\") pod \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") "
Oct 29 21:47:24.563304 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.563142    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-image-registry-private-configuration\") pod \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") "
Oct 29 21:47:24.563304 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.563195    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-trusted-ca\") pod \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") "
Oct 29 21:47:24.563304 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.563223    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-bound-sa-token\") pod \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\" (UID: \"81d7056d-8cf4-4058-a8fa-1731a0b93ff6\") "
Oct 29 21:47:24.563523 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.563474    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-certificates" pod "81d7056d-8cf4-4058-a8fa-1731a0b93ff6" (UID: "81d7056d-8cf4-4058-a8fa-1731a0b93ff6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:47:24.563768 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.563730    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-trusted-ca" pod "81d7056d-8cf4-4058-a8fa-1731a0b93ff6" (UID: "81d7056d-8cf4-4058-a8fa-1731a0b93ff6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:47:24.565647 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.565614    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-kube-api-access-xntk2" pod "81d7056d-8cf4-4058-a8fa-1731a0b93ff6" (UID: "81d7056d-8cf4-4058-a8fa-1731a0b93ff6"). InnerVolumeSpecName "kube-api-access-xntk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:47:24.565878 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.565854    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-installation-pull-secrets" pod "81d7056d-8cf4-4058-a8fa-1731a0b93ff6" (UID: "81d7056d-8cf4-4058-a8fa-1731a0b93ff6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:47:24.565967 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.565876    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-bound-sa-token" pod "81d7056d-8cf4-4058-a8fa-1731a0b93ff6" (UID: "81d7056d-8cf4-4058-a8fa-1731a0b93ff6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:47:24.565967 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.565909    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls" pod "81d7056d-8cf4-4058-a8fa-1731a0b93ff6" (UID: "81d7056d-8cf4-4058-a8fa-1731a0b93ff6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:47:24.566132 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.566113    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-image-registry-private-configuration" pod "81d7056d-8cf4-4058-a8fa-1731a0b93ff6" (UID: "81d7056d-8cf4-4058-a8fa-1731a0b93ff6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:47:24.570456 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.570427    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-ca-trust-extracted" pod "81d7056d-8cf4-4058-a8fa-1731a0b93ff6" (UID: "81d7056d-8cf4-4058-a8fa-1731a0b93ff6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:47:24.664258 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.664238    4962 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:47:24.664258 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.664258    4962 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-ca-trust-extracted\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:47:24.664405 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.664270    4962 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-image-registry-private-configuration\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:47:24.664405 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.664281    4962 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-trusted-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:47:24.664405 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.664311    4962 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-bound-sa-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:47:24.664405 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.664324    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xntk2\" (UniqueName: \"kubernetes.io/projected/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-kube-api-access-xntk2\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:47:24.664405 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.664337    4962 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-installation-pull-secrets\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:47:24.664405 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:24.664346    4962 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/81d7056d-8cf4-4058-a8fa-1731a0b93ff6-registry-certificates\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:47:25.417975 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:25.417942    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j" event={"ID":"81d7056d-8cf4-4058-a8fa-1731a0b93ff6","Type":"ContainerDied","Data":"665486255dfe48b35c60664417004e9dba8cd997383a3da17b0485c3ebb07e59"}
Oct 29 21:47:25.418376 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:25.418002    4962 scope.go:122] "RemoveContainer" containerID="4eeabd27b18f4b358ce07da09495036d184c136581e3837a802d98c897c55b29"
Oct 29 21:47:25.418376 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:25.417954    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55df9ddfdb-9gj4j"
Oct 29 21:47:25.484785 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:25.484764    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55df9ddfdb-9gj4j"]
Oct 29 21:47:25.496522 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:25.496489    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-55df9ddfdb-9gj4j"]
Oct 29 21:47:27.054789 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:27.054754    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81d7056d-8cf4-4058-a8fa-1731a0b93ff6" path="/var/lib/kubelet/pods/81d7056d-8cf4-4058-a8fa-1731a0b93ff6/volumes"
Oct 29 21:47:39.718730 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:47:39.718697    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:47:49.515888 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:49.515851    4962 generic.go:356] "Generic (PLEG): container finished" podID="30af126f-08a0-49e2-9988-50b2f8380cfe" containerID="36c03b38b676b32d5a3119ed532ca94892dd79907fecbdb13928cb375f8b8f49" exitCode=0
Oct 29 21:47:49.516336 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:49.515924    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2" event={"ID":"30af126f-08a0-49e2-9988-50b2f8380cfe","Type":"ContainerDied","Data":"36c03b38b676b32d5a3119ed532ca94892dd79907fecbdb13928cb375f8b8f49"}
Oct 29 21:47:49.516336 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:49.516193    4962 scope.go:122] "RemoveContainer" containerID="36c03b38b676b32d5a3119ed532ca94892dd79907fecbdb13928cb375f8b8f49"
Oct 29 21:47:50.521598 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:50.521559    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6h8t2" event={"ID":"30af126f-08a0-49e2-9988-50b2f8380cfe","Type":"ContainerStarted","Data":"687cb4462f0c33e47d03c7956eddd02a331609c9a633731c53ca9c79fe9d9687"}
Oct 29 21:47:50.523037 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:50.523015    4962 generic.go:356] "Generic (PLEG): container finished" podID="55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97" containerID="883f2e198a3926fa52689dc2aeaedbeb14c07e38b701b3ca6d1440f81cab02a3" exitCode=0
Oct 29 21:47:50.523141 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:50.523064    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-jtrf8" event={"ID":"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97","Type":"ContainerDied","Data":"883f2e198a3926fa52689dc2aeaedbeb14c07e38b701b3ca6d1440f81cab02a3"}
Oct 29 21:47:50.523339 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:50.523326    4962 scope.go:122] "RemoveContainer" containerID="883f2e198a3926fa52689dc2aeaedbeb14c07e38b701b3ca6d1440f81cab02a3"
Oct 29 21:47:51.080891 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.080838    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-p49l7-nlq2m"]
Oct 29 21:47:51.081062 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.081036    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-p49l7-nlq2m" podUID="0b0306f7-0104-4219-9fb4-a601014c7a43" containerName="insights-gathering" containerID="cri-o://35eb6d57641e98d1a75a7ce960c86ce2abd400b77dea64728c3443ab6bb9d034" gracePeriod=30
Oct 29 21:47:51.258207 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.255049    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-q6752-kglbr"]
Oct 29 21:47:51.258207 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.255470    4962 cpu_manager.go:414] "containerMap: removing container" podUID="81d7056d-8cf4-4058-a8fa-1731a0b93ff6" containerName="registry"
Oct 29 21:47:51.258207 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.255485    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81d7056d-8cf4-4058-a8fa-1731a0b93ff6" containerName="registry"
Oct 29 21:47:51.258207 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.255524    4962 cpu_manager.go:414] "containerMap: removing container" podUID="4b5fdb27-5000-4cf8-a025-13ae5389d17f" containerName="registry"
Oct 29 21:47:51.258207 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.255530    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b5fdb27-5000-4cf8-a025-13ae5389d17f" containerName="registry"
Oct 29 21:47:51.258207 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.255584    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b5fdb27-5000-4cf8-a025-13ae5389d17f" containerName="registry"
Oct 29 21:47:51.258207 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.255594    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="81d7056d-8cf4-4058-a8fa-1731a0b93ff6" containerName="registry"
Oct 29 21:47:51.258661 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.258420    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q6752-kglbr"
Oct 29 21:47:51.279241 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.279215    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-q6752-kglbr"]
Oct 29 21:47:51.329617 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.329589    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-p49l7-nlq2m"
Oct 29 21:47:51.346517 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.346465    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/66167c27-74ab-41b8-a549-f864497c1870-archives-path\") pod \"periodic-gathering-q6752-kglbr\" (UID: \"66167c27-74ab-41b8-a549-f864497c1870\") " pod="openshift-insights/periodic-gathering-q6752-kglbr"
Oct 29 21:47:51.346517 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.346498    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66167c27-74ab-41b8-a549-f864497c1870-service-ca-bundle\") pod \"periodic-gathering-q6752-kglbr\" (UID: \"66167c27-74ab-41b8-a549-f864497c1870\") " pod="openshift-insights/periodic-gathering-q6752-kglbr"
Oct 29 21:47:51.346662 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.346520    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66167c27-74ab-41b8-a549-f864497c1870-trusted-ca-bundle\") pod \"periodic-gathering-q6752-kglbr\" (UID: \"66167c27-74ab-41b8-a549-f864497c1870\") " pod="openshift-insights/periodic-gathering-q6752-kglbr"
Oct 29 21:47:51.346702 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.346654    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdndd\" (UniqueName: \"kubernetes.io/projected/66167c27-74ab-41b8-a549-f864497c1870-kube-api-access-tdndd\") pod \"periodic-gathering-q6752-kglbr\" (UID: \"66167c27-74ab-41b8-a549-f864497c1870\") " pod="openshift-insights/periodic-gathering-q6752-kglbr"
Oct 29 21:47:51.447618 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.447584    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b0306f7-0104-4219-9fb4-a601014c7a43-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0b0306f7-0104-4219-9fb4-a601014c7a43-archives-path\") pod \"0b0306f7-0104-4219-9fb4-a601014c7a43\" (UID: \"0b0306f7-0104-4219-9fb4-a601014c7a43\") "
Oct 29 21:47:51.447722 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.447624    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b0306f7-0104-4219-9fb4-a601014c7a43-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0306f7-0104-4219-9fb4-a601014c7a43-trusted-ca-bundle\") pod \"0b0306f7-0104-4219-9fb4-a601014c7a43\" (UID: \"0b0306f7-0104-4219-9fb4-a601014c7a43\") "
Oct 29 21:47:51.447722 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.447646    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b0306f7-0104-4219-9fb4-a601014c7a43-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0306f7-0104-4219-9fb4-a601014c7a43-service-ca-bundle\") pod \"0b0306f7-0104-4219-9fb4-a601014c7a43\" (UID: \"0b0306f7-0104-4219-9fb4-a601014c7a43\") "
Oct 29 21:47:51.447722 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.447672    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b0306f7-0104-4219-9fb4-a601014c7a43-kube-api-access-vdt58\" (UniqueName: \"kubernetes.io/projected/0b0306f7-0104-4219-9fb4-a601014c7a43-kube-api-access-vdt58\") pod \"0b0306f7-0104-4219-9fb4-a601014c7a43\" (UID: \"0b0306f7-0104-4219-9fb4-a601014c7a43\") "
Oct 29 21:47:51.447853 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.447757    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/66167c27-74ab-41b8-a549-f864497c1870-archives-path\") pod \"periodic-gathering-q6752-kglbr\" (UID: \"66167c27-74ab-41b8-a549-f864497c1870\") " pod="openshift-insights/periodic-gathering-q6752-kglbr"
Oct 29 21:47:51.447853 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.447790    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66167c27-74ab-41b8-a549-f864497c1870-service-ca-bundle\") pod \"periodic-gathering-q6752-kglbr\" (UID: \"66167c27-74ab-41b8-a549-f864497c1870\") " pod="openshift-insights/periodic-gathering-q6752-kglbr"
Oct 29 21:47:51.447853 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.447811    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66167c27-74ab-41b8-a549-f864497c1870-trusted-ca-bundle\") pod \"periodic-gathering-q6752-kglbr\" (UID: \"66167c27-74ab-41b8-a549-f864497c1870\") " pod="openshift-insights/periodic-gathering-q6752-kglbr"
Oct 29 21:47:51.447853 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.447832    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdndd\" (UniqueName: \"kubernetes.io/projected/66167c27-74ab-41b8-a549-f864497c1870-kube-api-access-tdndd\") pod \"periodic-gathering-q6752-kglbr\" (UID: \"66167c27-74ab-41b8-a549-f864497c1870\") " pod="openshift-insights/periodic-gathering-q6752-kglbr"
Oct 29 21:47:51.448055 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.447911    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0306f7-0104-4219-9fb4-a601014c7a43-archives-path" pod "0b0306f7-0104-4219-9fb4-a601014c7a43" (UID: "0b0306f7-0104-4219-9fb4-a601014c7a43"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:47:51.448055 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.448033    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0306f7-0104-4219-9fb4-a601014c7a43-service-ca-bundle" pod "0b0306f7-0104-4219-9fb4-a601014c7a43" (UID: "0b0306f7-0104-4219-9fb4-a601014c7a43"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:47:51.448152 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.448104    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0306f7-0104-4219-9fb4-a601014c7a43-trusted-ca-bundle" pod "0b0306f7-0104-4219-9fb4-a601014c7a43" (UID: "0b0306f7-0104-4219-9fb4-a601014c7a43"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:47:51.448152 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.448136    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/66167c27-74ab-41b8-a549-f864497c1870-archives-path\") pod \"periodic-gathering-q6752-kglbr\" (UID: \"66167c27-74ab-41b8-a549-f864497c1870\") " pod="openshift-insights/periodic-gathering-q6752-kglbr"
Oct 29 21:47:51.448484 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.448462    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66167c27-74ab-41b8-a549-f864497c1870-service-ca-bundle\") pod \"periodic-gathering-q6752-kglbr\" (UID: \"66167c27-74ab-41b8-a549-f864497c1870\") " pod="openshift-insights/periodic-gathering-q6752-kglbr"
Oct 29 21:47:51.448748 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.448731    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66167c27-74ab-41b8-a549-f864497c1870-trusted-ca-bundle\") pod \"periodic-gathering-q6752-kglbr\" (UID: \"66167c27-74ab-41b8-a549-f864497c1870\") " pod="openshift-insights/periodic-gathering-q6752-kglbr"
Oct 29 21:47:51.450001 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.449976    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0306f7-0104-4219-9fb4-a601014c7a43-kube-api-access-vdt58" pod "0b0306f7-0104-4219-9fb4-a601014c7a43" (UID: "0b0306f7-0104-4219-9fb4-a601014c7a43"). InnerVolumeSpecName "kube-api-access-vdt58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:47:51.471012 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.470992    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdndd\" (UniqueName: \"kubernetes.io/projected/66167c27-74ab-41b8-a549-f864497c1870-kube-api-access-tdndd\") pod \"periodic-gathering-q6752-kglbr\" (UID: \"66167c27-74ab-41b8-a549-f864497c1870\") " pod="openshift-insights/periodic-gathering-q6752-kglbr"
Oct 29 21:47:51.528706 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.528678    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-jtrf8" event={"ID":"55bbb5ff-1be8-4b7b-a12f-4dd1a3db7e97","Type":"ContainerStarted","Data":"b2a6ecf02081bfbe55337c552ab26989c70a10341880694e5da75617999e5fa6"}
Oct 29 21:47:51.529847 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.529825    4962 generic.go:356] "Generic (PLEG): container finished" podID="0b0306f7-0104-4219-9fb4-a601014c7a43" containerID="35eb6d57641e98d1a75a7ce960c86ce2abd400b77dea64728c3443ab6bb9d034" exitCode=2
Oct 29 21:47:51.529951 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.529869    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-p49l7-nlq2m"
Oct 29 21:47:51.530009 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.529871    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-p49l7-nlq2m" event={"ID":"0b0306f7-0104-4219-9fb4-a601014c7a43","Type":"ContainerDied","Data":"35eb6d57641e98d1a75a7ce960c86ce2abd400b77dea64728c3443ab6bb9d034"}
Oct 29 21:47:51.530009 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.529994    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-p49l7-nlq2m" event={"ID":"0b0306f7-0104-4219-9fb4-a601014c7a43","Type":"ContainerDied","Data":"2983dd1bcd7f3d015ff75d741a3741dbd5e61b6e4dab6a29d136e51acdac3325"}
Oct 29 21:47:51.530119 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.530021    4962 scope.go:122] "RemoveContainer" containerID="35eb6d57641e98d1a75a7ce960c86ce2abd400b77dea64728c3443ab6bb9d034"
Oct 29 21:47:51.541016 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.540990    4962 scope.go:122] "RemoveContainer" containerID="35eb6d57641e98d1a75a7ce960c86ce2abd400b77dea64728c3443ab6bb9d034"
Oct 29 21:47:51.541494 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:47:51.541473    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35eb6d57641e98d1a75a7ce960c86ce2abd400b77dea64728c3443ab6bb9d034\": container with ID starting with 35eb6d57641e98d1a75a7ce960c86ce2abd400b77dea64728c3443ab6bb9d034 not found: ID does not exist" containerID="35eb6d57641e98d1a75a7ce960c86ce2abd400b77dea64728c3443ab6bb9d034"
Oct 29 21:47:51.541604 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.541510    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35eb6d57641e98d1a75a7ce960c86ce2abd400b77dea64728c3443ab6bb9d034"} err="failed to get container status \"35eb6d57641e98d1a75a7ce960c86ce2abd400b77dea64728c3443ab6bb9d034\": rpc error: code = NotFound desc = could not find container \"35eb6d57641e98d1a75a7ce960c86ce2abd400b77dea64728c3443ab6bb9d034\": container with ID starting with 35eb6d57641e98d1a75a7ce960c86ce2abd400b77dea64728c3443ab6bb9d034 not found: ID does not exist"
Oct 29 21:47:51.548783 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.548765    4962 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0b0306f7-0104-4219-9fb4-a601014c7a43-archives-path\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:47:51.548783 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.548784    4962 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0306f7-0104-4219-9fb4-a601014c7a43-trusted-ca-bundle\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:47:51.548893 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.548794    4962 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0306f7-0104-4219-9fb4-a601014c7a43-service-ca-bundle\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:47:51.548893 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.548804    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdt58\" (UniqueName: \"kubernetes.io/projected/0b0306f7-0104-4219-9fb4-a601014c7a43-kube-api-access-vdt58\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:47:51.570677 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.570645    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q6752-kglbr"
Oct 29 21:47:51.607017 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.606942    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-p49l7-nlq2m"]
Oct 29 21:47:51.611040 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.611005    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-p49l7-nlq2m"]
Oct 29 21:47:51.875978 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:51.875903    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-q6752-kglbr"]
Oct 29 21:47:51.880027 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:47:51.879996    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66167c27_74ab_41b8_a549_f864497c1870.slice/crio-7084c2b10e83089f7e5d88baefed5c3750dba5977048018a24434ac6ddca19e3 WatchSource:0}: Error finding container 7084c2b10e83089f7e5d88baefed5c3750dba5977048018a24434ac6ddca19e3: Status 404 returned error can't find the container with id 7084c2b10e83089f7e5d88baefed5c3750dba5977048018a24434ac6ddca19e3
Oct 29 21:47:52.536332 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:52.536275    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q6752-kglbr" event={"ID":"66167c27-74ab-41b8-a549-f864497c1870","Type":"ContainerStarted","Data":"2bd92ff8aa44d59c5f22d50cbf0a50f799e033a75bb1f4de4c5efc6a8e9725bc"}
Oct 29 21:47:52.536804 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:52.536348    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q6752-kglbr" event={"ID":"66167c27-74ab-41b8-a549-f864497c1870","Type":"ContainerStarted","Data":"7084c2b10e83089f7e5d88baefed5c3750dba5977048018a24434ac6ddca19e3"}
Oct 29 21:47:52.579815 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:52.579774    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-q6752-kglbr" podStartSLOduration=1.579763726 podStartE2EDuration="1.579763726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:47:52.579083116 +0000 UTC m=+93.652040082" watchObservedRunningTime="2026-10-29 21:47:52.579763726 +0000 UTC m=+93.652720680"
Oct 29 21:47:53.055047 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:47:53.055012    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b0306f7-0104-4219-9fb4-a601014c7a43" path="/var/lib/kubelet/pods/0b0306f7-0104-4219-9fb4-a601014c7a43/volumes"
Oct 29 21:48:09.717480 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:48:09.717438    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:48:39.718768 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:48:39.718680    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:49:09.717413 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:49:09.717380    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:49:15.845882 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:15.845851    4962 generic.go:356] "Generic (PLEG): container finished" podID="66167c27-74ab-41b8-a549-f864497c1870" containerID="2bd92ff8aa44d59c5f22d50cbf0a50f799e033a75bb1f4de4c5efc6a8e9725bc" exitCode=0
Oct 29 21:49:15.846237 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:15.845889    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q6752-kglbr" event={"ID":"66167c27-74ab-41b8-a549-f864497c1870","Type":"ContainerDied","Data":"2bd92ff8aa44d59c5f22d50cbf0a50f799e033a75bb1f4de4c5efc6a8e9725bc"}
Oct 29 21:49:16.977964 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:16.977939    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q6752-kglbr"
Oct 29 21:49:17.035817 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:17.035794    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66167c27-74ab-41b8-a549-f864497c1870-kube-api-access-tdndd\" (UniqueName: \"kubernetes.io/projected/66167c27-74ab-41b8-a549-f864497c1870-kube-api-access-tdndd\") pod \"66167c27-74ab-41b8-a549-f864497c1870\" (UID: \"66167c27-74ab-41b8-a549-f864497c1870\") "
Oct 29 21:49:17.035928 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:17.035869    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66167c27-74ab-41b8-a549-f864497c1870-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66167c27-74ab-41b8-a549-f864497c1870-trusted-ca-bundle\") pod \"66167c27-74ab-41b8-a549-f864497c1870\" (UID: \"66167c27-74ab-41b8-a549-f864497c1870\") "
Oct 29 21:49:17.035928 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:17.035908    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66167c27-74ab-41b8-a549-f864497c1870-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/66167c27-74ab-41b8-a549-f864497c1870-archives-path\") pod \"66167c27-74ab-41b8-a549-f864497c1870\" (UID: \"66167c27-74ab-41b8-a549-f864497c1870\") "
Oct 29 21:49:17.036000 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:17.035945    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66167c27-74ab-41b8-a549-f864497c1870-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66167c27-74ab-41b8-a549-f864497c1870-service-ca-bundle\") pod \"66167c27-74ab-41b8-a549-f864497c1870\" (UID: \"66167c27-74ab-41b8-a549-f864497c1870\") "
Oct 29 21:49:17.036298 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:17.036216    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66167c27-74ab-41b8-a549-f864497c1870-archives-path" pod "66167c27-74ab-41b8-a549-f864497c1870" (UID: "66167c27-74ab-41b8-a549-f864497c1870"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:49:17.036357 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:17.036306    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66167c27-74ab-41b8-a549-f864497c1870-trusted-ca-bundle" pod "66167c27-74ab-41b8-a549-f864497c1870" (UID: "66167c27-74ab-41b8-a549-f864497c1870"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:49:17.036428 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:17.036400    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66167c27-74ab-41b8-a549-f864497c1870-service-ca-bundle" pod "66167c27-74ab-41b8-a549-f864497c1870" (UID: "66167c27-74ab-41b8-a549-f864497c1870"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:49:17.037866 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:17.037848    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66167c27-74ab-41b8-a549-f864497c1870-kube-api-access-tdndd" pod "66167c27-74ab-41b8-a549-f864497c1870" (UID: "66167c27-74ab-41b8-a549-f864497c1870"). InnerVolumeSpecName "kube-api-access-tdndd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:49:17.136586 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:17.136513    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdndd\" (UniqueName: \"kubernetes.io/projected/66167c27-74ab-41b8-a549-f864497c1870-kube-api-access-tdndd\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:49:17.136586 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:17.136543    4962 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66167c27-74ab-41b8-a549-f864497c1870-trusted-ca-bundle\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:49:17.136586 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:17.136554    4962 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/66167c27-74ab-41b8-a549-f864497c1870-archives-path\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:49:17.136586 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:17.136563    4962 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66167c27-74ab-41b8-a549-f864497c1870-service-ca-bundle\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:49:17.854830 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:17.854803    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-q6752-kglbr"
Oct 29 21:49:17.854830 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:17.854818    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-q6752-kglbr" event={"ID":"66167c27-74ab-41b8-a549-f864497c1870","Type":"ContainerDied","Data":"7084c2b10e83089f7e5d88baefed5c3750dba5977048018a24434ac6ddca19e3"}
Oct 29 21:49:17.855025 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:17.854864    4962 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7084c2b10e83089f7e5d88baefed5c3750dba5977048018a24434ac6ddca19e3"
Oct 29 21:49:21.613909 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.613879    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"]
Oct 29 21:49:21.614307 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.614184    4962 cpu_manager.go:414] "containerMap: removing container" podUID="66167c27-74ab-41b8-a549-f864497c1870" containerName="insights-gathering"
Oct 29 21:49:21.614307 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.614197    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66167c27-74ab-41b8-a549-f864497c1870" containerName="insights-gathering"
Oct 29 21:49:21.614307 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.614222    4962 cpu_manager.go:414] "containerMap: removing container" podUID="0b0306f7-0104-4219-9fb4-a601014c7a43" containerName="insights-gathering"
Oct 29 21:49:21.614307 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.614227    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b0306f7-0104-4219-9fb4-a601014c7a43" containerName="insights-gathering"
Oct 29 21:49:21.614307 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.614281    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b0306f7-0104-4219-9fb4-a601014c7a43" containerName="insights-gathering"
Oct 29 21:49:21.614307 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.614304    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="66167c27-74ab-41b8-a549-f864497c1870" containerName="insights-gathering"
Oct 29 21:49:21.617448 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.617428    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:21.632069 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.632050    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 29 21:49:21.632770 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.632752    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 29 21:49:21.643228 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.643205    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 29 21:49:21.643228 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.643223    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 29 21:49:21.643368 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.643270    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jjwwj\""
Oct 29 21:49:21.647075 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.647051    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"]
Oct 29 21:49:21.669202 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.669181    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-xpzqc\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:21.669318 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.669212    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fc110e07-7ec6-4455-9437-5b654b6cae44-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-xpzqc\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:21.669318 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.669246    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-xpzqc\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:21.669440 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.669412    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc110e07-7ec6-4455-9437-5b654b6cae44-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-xpzqc\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:21.669493 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.669453    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w768d\" (UniqueName: \"kubernetes.io/projected/fc110e07-7ec6-4455-9437-5b654b6cae44-kube-api-access-w768d\") pod \"istiod-openshift-gateway-7876bfcb74-xpzqc\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:21.669493 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.669485    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-xpzqc\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:21.669596 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.669576    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-xpzqc\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:21.770718 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.770698    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-xpzqc\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:21.770829 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.770733    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-xpzqc\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:21.770829 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.770755    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fc110e07-7ec6-4455-9437-5b654b6cae44-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-xpzqc\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:21.770829 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.770779    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-xpzqc\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:21.770829 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.770802    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc110e07-7ec6-4455-9437-5b654b6cae44-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-xpzqc\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:21.770829 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.770826    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w768d\" (UniqueName: \"kubernetes.io/projected/fc110e07-7ec6-4455-9437-5b654b6cae44-kube-api-access-w768d\") pod \"istiod-openshift-gateway-7876bfcb74-xpzqc\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:21.771067 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.770844    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-xpzqc\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:21.771405 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.771378    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-xpzqc\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:21.773100 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.773076    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc110e07-7ec6-4455-9437-5b654b6cae44-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-xpzqc\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:21.773100 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.773085    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-xpzqc\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:21.773244 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.773106    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-xpzqc\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:21.773244 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.773194    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fc110e07-7ec6-4455-9437-5b654b6cae44-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-xpzqc\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:21.795090 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.795070    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-xpzqc\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:21.795892 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.795872    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w768d\" (UniqueName: \"kubernetes.io/projected/fc110e07-7ec6-4455-9437-5b654b6cae44-kube-api-access-w768d\") pod \"istiod-openshift-gateway-7876bfcb74-xpzqc\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:21.927710 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:21.927686    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:22.237155 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:22.237070    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"]
Oct 29 21:49:22.240019 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:49:22.239990    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc110e07_7ec6_4455_9437_5b654b6cae44.slice/crio-5404f6e6de1d8660055c0920e674166b9989f3aa746439b0ce480bd4a92642a7 WatchSource:0}: Error finding container 5404f6e6de1d8660055c0920e674166b9989f3aa746439b0ce480bd4a92642a7: Status 404 returned error can't find the container with id 5404f6e6de1d8660055c0920e674166b9989f3aa746439b0ce480bd4a92642a7
Oct 29 21:49:22.873829 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:22.873798    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc" event={"ID":"fc110e07-7ec6-4455-9437-5b654b6cae44","Type":"ContainerStarted","Data":"5404f6e6de1d8660055c0920e674166b9989f3aa746439b0ce480bd4a92642a7"}
Oct 29 21:49:24.809222 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:24.809191    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:49:24.809498 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:24.809254    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:49:24.882933 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:24.882902    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc" event={"ID":"fc110e07-7ec6-4455-9437-5b654b6cae44","Type":"ContainerStarted","Data":"5e68e3b2233253355cda330a1366d9bfa4f1965396b9362b9da19691b7a8aa97"}
Oct 29 21:49:24.883102 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:24.883080    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:24.936783 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:24.936737    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc" podStartSLOduration=1.370264418 podStartE2EDuration="3.936723525s" totalImagesPullingTime="2.566459107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:49:24.935754427 +0000 UTC m=+186.008711392" watchObservedRunningTime="2026-10-29 21:49:24.936723525 +0000 UTC m=+186.009680489"
Oct 29 21:49:25.889107 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:25.889066    4962 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-xpzqc container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 21:49:25.889488 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:25.889127    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc" podUID="fc110e07-7ec6-4455-9437-5b654b6cae44" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 21:49:28.888694 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:28.888666    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:49:39.717626 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:49:39.717593    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:49:51.473864 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.473830    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"]
Oct 29 21:49:51.477355 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.477339    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"
Oct 29 21:49:51.509497 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.509467    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"]
Oct 29 21:49:51.552878 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.552852    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 29 21:49:51.553070 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.553061    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 29 21:49:51.553134 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.553114    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 29 21:49:51.553177 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.553115    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 29 21:49:51.553177 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.553140    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 29 21:49:51.553246 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.553185    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-6rlhk\""
Oct 29 21:49:51.580488 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.580467    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgqkl\" (UniqueName: \"kubernetes.io/projected/67ce20a8-b6b0-49db-8d7a-abdaad41de45-kube-api-access-jgqkl\") pod \"opendatahub-operator-controller-manager-59b57f6f78-g9k4n\" (UID: \"67ce20a8-b6b0-49db-8d7a-abdaad41de45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"
Oct 29 21:49:51.580564 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.580502    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/67ce20a8-b6b0-49db-8d7a-abdaad41de45-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-g9k4n\" (UID: \"67ce20a8-b6b0-49db-8d7a-abdaad41de45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"
Oct 29 21:49:51.580564 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.580535    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67ce20a8-b6b0-49db-8d7a-abdaad41de45-webhook-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-g9k4n\" (UID: \"67ce20a8-b6b0-49db-8d7a-abdaad41de45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"
Oct 29 21:49:51.580667 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.580646    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/67ce20a8-b6b0-49db-8d7a-abdaad41de45-metrics-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-g9k4n\" (UID: \"67ce20a8-b6b0-49db-8d7a-abdaad41de45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"
Oct 29 21:49:51.580725 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.580713    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/67ce20a8-b6b0-49db-8d7a-abdaad41de45-manifests\") pod \"opendatahub-operator-controller-manager-59b57f6f78-g9k4n\" (UID: \"67ce20a8-b6b0-49db-8d7a-abdaad41de45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"
Oct 29 21:49:51.580764 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.580732    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67ce20a8-b6b0-49db-8d7a-abdaad41de45-tmp\") pod \"opendatahub-operator-controller-manager-59b57f6f78-g9k4n\" (UID: \"67ce20a8-b6b0-49db-8d7a-abdaad41de45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"
Oct 29 21:49:51.681811 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.681775    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67ce20a8-b6b0-49db-8d7a-abdaad41de45-webhook-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-g9k4n\" (UID: \"67ce20a8-b6b0-49db-8d7a-abdaad41de45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"
Oct 29 21:49:51.681995 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.681815    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/67ce20a8-b6b0-49db-8d7a-abdaad41de45-metrics-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-g9k4n\" (UID: \"67ce20a8-b6b0-49db-8d7a-abdaad41de45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"
Oct 29 21:49:51.681995 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.681875    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/67ce20a8-b6b0-49db-8d7a-abdaad41de45-manifests\") pod \"opendatahub-operator-controller-manager-59b57f6f78-g9k4n\" (UID: \"67ce20a8-b6b0-49db-8d7a-abdaad41de45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"
Oct 29 21:49:51.681995 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.681899    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67ce20a8-b6b0-49db-8d7a-abdaad41de45-tmp\") pod \"opendatahub-operator-controller-manager-59b57f6f78-g9k4n\" (UID: \"67ce20a8-b6b0-49db-8d7a-abdaad41de45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"
Oct 29 21:49:51.681995 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.681930    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgqkl\" (UniqueName: \"kubernetes.io/projected/67ce20a8-b6b0-49db-8d7a-abdaad41de45-kube-api-access-jgqkl\") pod \"opendatahub-operator-controller-manager-59b57f6f78-g9k4n\" (UID: \"67ce20a8-b6b0-49db-8d7a-abdaad41de45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"
Oct 29 21:49:51.681995 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.681967    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/67ce20a8-b6b0-49db-8d7a-abdaad41de45-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-g9k4n\" (UID: \"67ce20a8-b6b0-49db-8d7a-abdaad41de45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"
Oct 29 21:49:51.682447 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.682420    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/67ce20a8-b6b0-49db-8d7a-abdaad41de45-manifests\") pod \"opendatahub-operator-controller-manager-59b57f6f78-g9k4n\" (UID: \"67ce20a8-b6b0-49db-8d7a-abdaad41de45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"
Oct 29 21:49:51.684472 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.684449    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/67ce20a8-b6b0-49db-8d7a-abdaad41de45-tmp\") pod \"opendatahub-operator-controller-manager-59b57f6f78-g9k4n\" (UID: \"67ce20a8-b6b0-49db-8d7a-abdaad41de45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"
Oct 29 21:49:51.684600 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.684582    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/67ce20a8-b6b0-49db-8d7a-abdaad41de45-apiservice-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-g9k4n\" (UID: \"67ce20a8-b6b0-49db-8d7a-abdaad41de45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"
Oct 29 21:49:51.684772 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.684751    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/67ce20a8-b6b0-49db-8d7a-abdaad41de45-metrics-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-g9k4n\" (UID: \"67ce20a8-b6b0-49db-8d7a-abdaad41de45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"
Oct 29 21:49:51.684884 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.684865    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/67ce20a8-b6b0-49db-8d7a-abdaad41de45-webhook-cert\") pod \"opendatahub-operator-controller-manager-59b57f6f78-g9k4n\" (UID: \"67ce20a8-b6b0-49db-8d7a-abdaad41de45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"
Oct 29 21:49:51.705654 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.705630    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgqkl\" (UniqueName: \"kubernetes.io/projected/67ce20a8-b6b0-49db-8d7a-abdaad41de45-kube-api-access-jgqkl\") pod \"opendatahub-operator-controller-manager-59b57f6f78-g9k4n\" (UID: \"67ce20a8-b6b0-49db-8d7a-abdaad41de45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"
Oct 29 21:49:51.792771 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:51.792695    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"
Oct 29 21:49:52.142494 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:52.142467    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"]
Oct 29 21:49:52.145151 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:49:52.145124    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ce20a8_b6b0_49db_8d7a_abdaad41de45.slice/crio-17015377bf4e68c7cfd4f088847752d343a1ac3a7f99e3dbd14e46a9b574b3f2 WatchSource:0}: Error finding container 17015377bf4e68c7cfd4f088847752d343a1ac3a7f99e3dbd14e46a9b574b3f2: Status 404 returned error can't find the container with id 17015377bf4e68c7cfd4f088847752d343a1ac3a7f99e3dbd14e46a9b574b3f2
Oct 29 21:49:52.994067 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:52.993995    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n" event={"ID":"67ce20a8-b6b0-49db-8d7a-abdaad41de45","Type":"ContainerStarted","Data":"17015377bf4e68c7cfd4f088847752d343a1ac3a7f99e3dbd14e46a9b574b3f2"}
Oct 29 21:49:55.004569 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:55.004527    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n" event={"ID":"67ce20a8-b6b0-49db-8d7a-abdaad41de45","Type":"ContainerStarted","Data":"7fe41a6bed388529fc4ae011704e2e6bf26f8a6a185729166b355e5cf5f1ba11"}
Oct 29 21:49:56.010129 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:56.010093    4962 generic.go:356] "Generic (PLEG): container finished" podID="67ce20a8-b6b0-49db-8d7a-abdaad41de45" containerID="7fe41a6bed388529fc4ae011704e2e6bf26f8a6a185729166b355e5cf5f1ba11" exitCode=0
Oct 29 21:49:56.010518 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:56.010181    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n" event={"ID":"67ce20a8-b6b0-49db-8d7a-abdaad41de45","Type":"ContainerDied","Data":"7fe41a6bed388529fc4ae011704e2e6bf26f8a6a185729166b355e5cf5f1ba11"}
Oct 29 21:49:57.015875 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:57.015845    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n" event={"ID":"67ce20a8-b6b0-49db-8d7a-abdaad41de45","Type":"ContainerStarted","Data":"a74684c707c26f1cda88b1d97e5b88da4e87a1805e62ca5af3af1c1ddce79c55"}
Oct 29 21:49:57.016275 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:57.015947    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"
Oct 29 21:49:57.107542 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:49:57.107497    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n" podStartSLOduration=2.60987709 podStartE2EDuration="6.107483328s" totalImagesPullingTime="3.012948444s" totalInitContainerRuntime="484.657794ms" isStatefulPod=false podCreationTimestamp="2026-10-29 21:49:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:49:57.094617277 +0000 UTC m=+218.167574280" watchObservedRunningTime="2026-10-29 21:49:57.107483328 +0000 UTC m=+218.180440291"
Oct 29 21:50:08.022011 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:08.021936    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-59b57f6f78-g9k4n"
Oct 29 21:50:09.719161 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:50:09.719128    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:50:28.728492 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:28.728462    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-68669df9c4-ktqlt"]
Oct 29 21:50:28.794777 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:28.794754    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-68669df9c4-ktqlt"]
Oct 29 21:50:28.794923 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:28.794862    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-ktqlt"
Oct 29 21:50:28.820727 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:28.820698    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-p42kq\""
Oct 29 21:50:28.820848 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:28.820761    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 29 21:50:28.820848 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:28.820775    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 29 21:50:28.844688 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:28.844667    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqjtx\" (UniqueName: \"kubernetes.io/projected/3391c781-0e29-49eb-a190-20229a0ab066-kube-api-access-vqjtx\") pod \"limitador-operator-controller-manager-68669df9c4-ktqlt\" (UID: \"3391c781-0e29-49eb-a190-20229a0ab066\") " pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-ktqlt"
Oct 29 21:50:28.945762 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:28.945736    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqjtx\" (UniqueName: \"kubernetes.io/projected/3391c781-0e29-49eb-a190-20229a0ab066-kube-api-access-vqjtx\") pod \"limitador-operator-controller-manager-68669df9c4-ktqlt\" (UID: \"3391c781-0e29-49eb-a190-20229a0ab066\") " pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-ktqlt"
Oct 29 21:50:28.969015 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:28.968993    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqjtx\" (UniqueName: \"kubernetes.io/projected/3391c781-0e29-49eb-a190-20229a0ab066-kube-api-access-vqjtx\") pod \"limitador-operator-controller-manager-68669df9c4-ktqlt\" (UID: \"3391c781-0e29-49eb-a190-20229a0ab066\") " pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-ktqlt"
Oct 29 21:50:29.121190 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:29.121117    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-ktqlt"
Oct 29 21:50:29.417921 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:29.417787    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-68669df9c4-ktqlt"]
Oct 29 21:50:29.420467 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:50:29.420437    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3391c781_0e29_49eb_a190_20229a0ab066.slice/crio-ce58d9222e38dfae0eaf553febfc528f99cb668eef655d1922f1f0fcf021fc41 WatchSource:0}: Error finding container ce58d9222e38dfae0eaf553febfc528f99cb668eef655d1922f1f0fcf021fc41: Status 404 returned error can't find the container with id ce58d9222e38dfae0eaf553febfc528f99cb668eef655d1922f1f0fcf021fc41
Oct 29 21:50:30.140970 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:30.140934    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-ktqlt" event={"ID":"3391c781-0e29-49eb-a190-20229a0ab066","Type":"ContainerStarted","Data":"ce58d9222e38dfae0eaf553febfc528f99cb668eef655d1922f1f0fcf021fc41"}
Oct 29 21:50:30.499262 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:30.499230    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-92hd8"]
Oct 29 21:50:30.503180 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:30.502537    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-92hd8"
Oct 29 21:50:30.517487 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:30.517462    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8846b\""
Oct 29 21:50:30.517601 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:30.517512    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 29 21:50:30.529773 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:30.529750    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-92hd8"]
Oct 29 21:50:30.557203 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:30.557176    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac207213-067e-4673-b4b2-630469c878a8-plugin-serving-cert\") pod \"kuadrant-console-plugin-7bbc6bb9d7-92hd8\" (UID: \"ac207213-067e-4673-b4b2-630469c878a8\") " pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-92hd8"
Oct 29 21:50:30.557348 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:30.557232    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhk4k\" (UniqueName: \"kubernetes.io/projected/ac207213-067e-4673-b4b2-630469c878a8-kube-api-access-bhk4k\") pod \"kuadrant-console-plugin-7bbc6bb9d7-92hd8\" (UID: \"ac207213-067e-4673-b4b2-630469c878a8\") " pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-92hd8"
Oct 29 21:50:30.657993 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:30.657956    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac207213-067e-4673-b4b2-630469c878a8-plugin-serving-cert\") pod \"kuadrant-console-plugin-7bbc6bb9d7-92hd8\" (UID: \"ac207213-067e-4673-b4b2-630469c878a8\") " pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-92hd8"
Oct 29 21:50:30.658155 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:30.658023    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhk4k\" (UniqueName: \"kubernetes.io/projected/ac207213-067e-4673-b4b2-630469c878a8-kube-api-access-bhk4k\") pod \"kuadrant-console-plugin-7bbc6bb9d7-92hd8\" (UID: \"ac207213-067e-4673-b4b2-630469c878a8\") " pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-92hd8"
Oct 29 21:50:30.660812 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:30.660782    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ac207213-067e-4673-b4b2-630469c878a8-plugin-serving-cert\") pod \"kuadrant-console-plugin-7bbc6bb9d7-92hd8\" (UID: \"ac207213-067e-4673-b4b2-630469c878a8\") " pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-92hd8"
Oct 29 21:50:30.684474 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:30.684447    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhk4k\" (UniqueName: \"kubernetes.io/projected/ac207213-067e-4673-b4b2-630469c878a8-kube-api-access-bhk4k\") pod \"kuadrant-console-plugin-7bbc6bb9d7-92hd8\" (UID: \"ac207213-067e-4673-b4b2-630469c878a8\") " pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-92hd8"
Oct 29 21:50:30.826728 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:30.826646    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-92hd8"
Oct 29 21:50:31.130806 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:31.130740    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-92hd8"]
Oct 29 21:50:31.134917 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:50:31.134882    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac207213_067e_4673_b4b2_630469c878a8.slice/crio-b88a62dbc44073b2ac6ac0c7ccf864580f6ad81c0a8572bcd67fc4048941f53e WatchSource:0}: Error finding container b88a62dbc44073b2ac6ac0c7ccf864580f6ad81c0a8572bcd67fc4048941f53e: Status 404 returned error can't find the container with id b88a62dbc44073b2ac6ac0c7ccf864580f6ad81c0a8572bcd67fc4048941f53e
Oct 29 21:50:31.147428 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:31.147397    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-92hd8" event={"ID":"ac207213-067e-4673-b4b2-630469c878a8","Type":"ContainerStarted","Data":"b88a62dbc44073b2ac6ac0c7ccf864580f6ad81c0a8572bcd67fc4048941f53e"}
Oct 29 21:50:32.153510 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:32.153425    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-ktqlt" event={"ID":"3391c781-0e29-49eb-a190-20229a0ab066","Type":"ContainerStarted","Data":"ff145e63029f33dabc44bb81823f45056f2f08f7dfa39878eafd2c7772192ab0"}
Oct 29 21:50:32.153884 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:32.153568    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-ktqlt"
Oct 29 21:50:32.194232 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:32.194185    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-ktqlt" podStartSLOduration=1.74003073 podStartE2EDuration="4.194170964s" totalImagesPullingTime="2.454140234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:50:32.193569272 +0000 UTC m=+253.266526249" watchObservedRunningTime="2026-10-29 21:50:32.194170964 +0000 UTC m=+253.267127932"
Oct 29 21:50:33.158973 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:33.158936    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-92hd8" event={"ID":"ac207213-067e-4673-b4b2-630469c878a8","Type":"ContainerStarted","Data":"4331b6802d528d731c972c5b989f8fa07657296089d4946325aaa7db1a2f6572"}
Oct 29 21:50:33.203144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:33.203101    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-7bbc6bb9d7-92hd8" podStartSLOduration=1.984654267 podStartE2EDuration="3.203088784s" totalImagesPullingTime="1.218434517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:50:33.202923433 +0000 UTC m=+254.275880396" watchObservedRunningTime="2026-10-29 21:50:33.203088784 +0000 UTC m=+254.276045748"
Oct 29 21:50:39.717722 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:50:39.717688    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:50:41.461084 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:41.461050    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-64db8fcdf6-qnr64"]
Oct 29 21:50:41.464510 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:41.464488    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-64db8fcdf6-qnr64"
Oct 29 21:50:41.479271 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:41.479251    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 29 21:50:41.490881 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:41.490854    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 29 21:50:41.491003 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:41.490911    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 29 21:50:41.491003 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:41.490965    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 29 21:50:41.491125 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:41.491033    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-75b62\""
Oct 29 21:50:41.491125 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:41.491034    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 29 21:50:41.492222 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:41.492202    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-64db8fcdf6-qnr64"]
Oct 29 21:50:41.541551 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:41.541528    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0-metrics-certs\") pod \"jobset-controller-manager-64db8fcdf6-qnr64\" (UID: \"ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0\") " pod="openshift-jobset-operator/jobset-controller-manager-64db8fcdf6-qnr64"
Oct 29 21:50:41.541649 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:41.541562    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0-manager-config\") pod \"jobset-controller-manager-64db8fcdf6-qnr64\" (UID: \"ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0\") " pod="openshift-jobset-operator/jobset-controller-manager-64db8fcdf6-qnr64"
Oct 29 21:50:41.541649 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:41.541583    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0-cert\") pod \"jobset-controller-manager-64db8fcdf6-qnr64\" (UID: \"ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0\") " pod="openshift-jobset-operator/jobset-controller-manager-64db8fcdf6-qnr64"
Oct 29 21:50:41.541723 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:41.541697    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wxqc\" (UniqueName: \"kubernetes.io/projected/ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0-kube-api-access-7wxqc\") pod \"jobset-controller-manager-64db8fcdf6-qnr64\" (UID: \"ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0\") " pod="openshift-jobset-operator/jobset-controller-manager-64db8fcdf6-qnr64"
Oct 29 21:50:41.642440 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:41.642409    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wxqc\" (UniqueName: \"kubernetes.io/projected/ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0-kube-api-access-7wxqc\") pod \"jobset-controller-manager-64db8fcdf6-qnr64\" (UID: \"ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0\") " pod="openshift-jobset-operator/jobset-controller-manager-64db8fcdf6-qnr64"
Oct 29 21:50:41.642529 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:41.642470    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0-metrics-certs\") pod \"jobset-controller-manager-64db8fcdf6-qnr64\" (UID: \"ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0\") " pod="openshift-jobset-operator/jobset-controller-manager-64db8fcdf6-qnr64"
Oct 29 21:50:41.642529 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:41.642493    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0-manager-config\") pod \"jobset-controller-manager-64db8fcdf6-qnr64\" (UID: \"ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0\") " pod="openshift-jobset-operator/jobset-controller-manager-64db8fcdf6-qnr64"
Oct 29 21:50:41.642529 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:41.642515    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0-cert\") pod \"jobset-controller-manager-64db8fcdf6-qnr64\" (UID: \"ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0\") " pod="openshift-jobset-operator/jobset-controller-manager-64db8fcdf6-qnr64"
Oct 29 21:50:41.643232 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:41.643213    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0-manager-config\") pod \"jobset-controller-manager-64db8fcdf6-qnr64\" (UID: \"ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0\") " pod="openshift-jobset-operator/jobset-controller-manager-64db8fcdf6-qnr64"
Oct 29 21:50:41.644980 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:41.644955    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0-cert\") pod \"jobset-controller-manager-64db8fcdf6-qnr64\" (UID: \"ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0\") " pod="openshift-jobset-operator/jobset-controller-manager-64db8fcdf6-qnr64"
Oct 29 21:50:41.645063 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:41.645043    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0-metrics-certs\") pod \"jobset-controller-manager-64db8fcdf6-qnr64\" (UID: \"ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0\") " pod="openshift-jobset-operator/jobset-controller-manager-64db8fcdf6-qnr64"
Oct 29 21:50:41.665173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:41.665149    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wxqc\" (UniqueName: \"kubernetes.io/projected/ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0-kube-api-access-7wxqc\") pod \"jobset-controller-manager-64db8fcdf6-qnr64\" (UID: \"ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0\") " pod="openshift-jobset-operator/jobset-controller-manager-64db8fcdf6-qnr64"
Oct 29 21:50:41.775371 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:41.775322    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-64db8fcdf6-qnr64"
Oct 29 21:50:42.069883 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:42.069819    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-64db8fcdf6-qnr64"]
Oct 29 21:50:42.072745 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:50:42.072722    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1316d3_8ab8_47a6_9082_5ad65f5a6eb0.slice/crio-f374d9b6adf7ed5927b10be74972f7b4ceaa1329941b9388f3f396049a3a275a WatchSource:0}: Error finding container f374d9b6adf7ed5927b10be74972f7b4ceaa1329941b9388f3f396049a3a275a: Status 404 returned error can't find the container with id f374d9b6adf7ed5927b10be74972f7b4ceaa1329941b9388f3f396049a3a275a
Oct 29 21:50:42.195466 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:42.195429    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-64db8fcdf6-qnr64" event={"ID":"ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0","Type":"ContainerStarted","Data":"f374d9b6adf7ed5927b10be74972f7b4ceaa1329941b9388f3f396049a3a275a"}
Oct 29 21:50:43.161459 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:43.161425    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-68669df9c4-ktqlt"
Oct 29 21:50:45.210589 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:45.210554    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-64db8fcdf6-qnr64" event={"ID":"ae1316d3-8ab8-47a6-9082-5ad65f5a6eb0","Type":"ContainerStarted","Data":"76134eaec572af9d315b7395039afcebcb72ff818238c395178da44534fd296c"}
Oct 29 21:50:45.210958 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:45.210626    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-64db8fcdf6-qnr64"
Oct 29 21:50:45.252083 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:45.252032    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-64db8fcdf6-qnr64" podStartSLOduration=1.875374001 podStartE2EDuration="4.25201912s" totalImagesPullingTime="2.376645119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:50:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:50:45.251215638 +0000 UTC m=+266.324172619" watchObservedRunningTime="2026-10-29 21:50:45.25201912 +0000 UTC m=+266.324976084"
Oct 29 21:50:56.219734 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:50:56.219698    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-64db8fcdf6-qnr64"
Oct 29 21:51:09.718397 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:51:09.718356    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:51:19.071496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:19.071468    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/2.log"
Oct 29 21:51:19.073527 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:19.073507    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/2.log"
Oct 29 21:51:19.079086 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:19.079066    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74bdq_259e6b95-d64c-4101-871f-4a933d4ac38f/ovn-acl-logging/0.log"
Oct 29 21:51:19.080770 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:19.080743    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74bdq_259e6b95-d64c-4101-871f-4a933d4ac38f/ovn-acl-logging/0.log"
Oct 29 21:51:27.479166 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.479134    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"]
Oct 29 21:51:27.488509 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.488486    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:27.512098 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.512071    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"]
Oct 29 21:51:27.583333 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.583302    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rnqn\" (UniqueName: \"kubernetes.io/projected/2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5-kube-api-access-7rnqn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bbmrg\" (UID: \"2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:27.583500 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.583348    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bbmrg\" (UID: \"2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:27.583578 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.583502    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bbmrg\" (UID: \"2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:27.583578 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.583536    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bbmrg\" (UID: \"2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:27.583654 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.583611    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bbmrg\" (UID: \"2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:27.583707 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.583655    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bbmrg\" (UID: \"2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:27.583771 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.583708    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bbmrg\" (UID: \"2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:27.684761 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.684733    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bbmrg\" (UID: \"2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:27.684897 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.684769    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bbmrg\" (UID: \"2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:27.684897 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.684784    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bbmrg\" (UID: \"2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:27.684897 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.684800    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bbmrg\" (UID: \"2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:27.684897 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.684820    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bbmrg\" (UID: \"2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:27.684897 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.684835    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bbmrg\" (UID: \"2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:27.684897 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.684884    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rnqn\" (UniqueName: \"kubernetes.io/projected/2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5-kube-api-access-7rnqn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bbmrg\" (UID: \"2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:27.685864 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.685833    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bbmrg\" (UID: \"2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:27.687154 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.687125    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bbmrg\" (UID: \"2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:27.687281 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.687261    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bbmrg\" (UID: \"2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:27.687357 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.687332    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bbmrg\" (UID: \"2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:27.687417 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.687387    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bbmrg\" (UID: \"2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:27.723424 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.723406    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bbmrg\" (UID: \"2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:27.725115 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.725088    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rnqn\" (UniqueName: \"kubernetes.io/projected/2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5-kube-api-access-7rnqn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bbmrg\" (UID: \"2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:27.799653 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:27.799601    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:28.112109 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:28.112081    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"]
Oct 29 21:51:28.114606 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:51:28.114573    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5c68ae_59b4_43ab_a8a6_cbfb407d51f5.slice/crio-ddbfc5e00531935fa03c5c6d890c3ae046ebf730223dd11ff564a144f3d36fde WatchSource:0}: Error finding container ddbfc5e00531935fa03c5c6d890c3ae046ebf730223dd11ff564a144f3d36fde: Status 404 returned error can't find the container with id ddbfc5e00531935fa03c5c6d890c3ae046ebf730223dd11ff564a144f3d36fde
Oct 29 21:51:28.116642 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:28.116611    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:51:28.116701 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:28.116675    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:51:28.390548 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:28.390454    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg" event={"ID":"2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5","Type":"ContainerStarted","Data":"4f0254139588d419acaf54e5bd677264ccd8584937d746ec4e93f65e4d9fe38e"}
Oct 29 21:51:28.390548 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:28.390514    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg" event={"ID":"2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5","Type":"ContainerStarted","Data":"ddbfc5e00531935fa03c5c6d890c3ae046ebf730223dd11ff564a144f3d36fde"}
Oct 29 21:51:28.390770 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:28.390579    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:28.443626 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:28.443566    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg" podStartSLOduration=1.443548326 podStartE2EDuration="1.443548326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:51:28.436562977 +0000 UTC m=+309.509519942" watchObservedRunningTime="2026-10-29 21:51:28.443548326 +0000 UTC m=+309.516505291"
Oct 29 21:51:29.396608 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:29.396582    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bbmrg"
Oct 29 21:51:29.490063 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:29.490033    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"]
Oct 29 21:51:29.490258 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:29.490224    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc" podUID="fc110e07-7ec6-4455-9437-5b654b6cae44" containerName="discovery" containerID="cri-o://5e68e3b2233253355cda330a1366d9bfa4f1965396b9362b9da19691b7a8aa97" gracePeriod=30
Oct 29 21:51:29.728751 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:29.728729    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:51:29.903464 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:29.903422    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc110e07-7ec6-4455-9437-5b654b6cae44-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc110e07-7ec6-4455-9437-5b654b6cae44-local-certs\") pod \"fc110e07-7ec6-4455-9437-5b654b6cae44\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") "
Oct 29 21:51:29.903630 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:29.903514    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-csr-dns-cert\") pod \"fc110e07-7ec6-4455-9437-5b654b6cae44\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") "
Oct 29 21:51:29.903630 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:29.903564    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-token\" (UniqueName: \"kubernetes.io/projected/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-token\") pod \"fc110e07-7ec6-4455-9437-5b654b6cae44\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") "
Oct 29 21:51:29.903736 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:29.903695    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc110e07-7ec6-4455-9437-5b654b6cae44-cacerts\" (UniqueName: \"kubernetes.io/secret/fc110e07-7ec6-4455-9437-5b654b6cae44-cacerts\") pod \"fc110e07-7ec6-4455-9437-5b654b6cae44\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") "
Oct 29 21:51:29.903784 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:29.903771    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-csr-ca-configmap\") pod \"fc110e07-7ec6-4455-9437-5b654b6cae44\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") "
Oct 29 21:51:29.903868 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:29.903828    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-kubeconfig\") pod \"fc110e07-7ec6-4455-9437-5b654b6cae44\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") "
Oct 29 21:51:29.903939 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:29.903863    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc110e07-7ec6-4455-9437-5b654b6cae44-kube-api-access-w768d\" (UniqueName: \"kubernetes.io/projected/fc110e07-7ec6-4455-9437-5b654b6cae44-kube-api-access-w768d\") pod \"fc110e07-7ec6-4455-9437-5b654b6cae44\" (UID: \"fc110e07-7ec6-4455-9437-5b654b6cae44\") "
Oct 29 21:51:29.904181 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:29.904151    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-csr-ca-configmap" pod "fc110e07-7ec6-4455-9437-5b654b6cae44" (UID: "fc110e07-7ec6-4455-9437-5b654b6cae44"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:51:29.906229 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:29.906162    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-token" pod "fc110e07-7ec6-4455-9437-5b654b6cae44" (UID: "fc110e07-7ec6-4455-9437-5b654b6cae44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:51:29.906330 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:29.906240    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-csr-dns-cert" pod "fc110e07-7ec6-4455-9437-5b654b6cae44" (UID: "fc110e07-7ec6-4455-9437-5b654b6cae44"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:51:29.906330 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:29.906279    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc110e07-7ec6-4455-9437-5b654b6cae44-cacerts" pod "fc110e07-7ec6-4455-9437-5b654b6cae44" (UID: "fc110e07-7ec6-4455-9437-5b654b6cae44"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:51:29.906500 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:29.906484    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc110e07-7ec6-4455-9437-5b654b6cae44-local-certs" pod "fc110e07-7ec6-4455-9437-5b654b6cae44" (UID: "fc110e07-7ec6-4455-9437-5b654b6cae44"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:51:29.906729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:29.906710    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-kubeconfig" pod "fc110e07-7ec6-4455-9437-5b654b6cae44" (UID: "fc110e07-7ec6-4455-9437-5b654b6cae44"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:51:29.906779 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:29.906757    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc110e07-7ec6-4455-9437-5b654b6cae44-kube-api-access-w768d" pod "fc110e07-7ec6-4455-9437-5b654b6cae44" (UID: "fc110e07-7ec6-4455-9437-5b654b6cae44"). InnerVolumeSpecName "kube-api-access-w768d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:51:30.005052 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:30.005020    4962 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-kubeconfig\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:51:30.005052 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:30.005050    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w768d\" (UniqueName: \"kubernetes.io/projected/fc110e07-7ec6-4455-9437-5b654b6cae44-kube-api-access-w768d\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:51:30.005052 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:30.005061    4962 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc110e07-7ec6-4455-9437-5b654b6cae44-local-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:51:30.005206 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:30.005071    4962 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-csr-dns-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:51:30.005206 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:30.005079    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:51:30.005206 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:30.005087    4962 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fc110e07-7ec6-4455-9437-5b654b6cae44-cacerts\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:51:30.005206 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:30.005095    4962 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fc110e07-7ec6-4455-9437-5b654b6cae44-istio-csr-ca-configmap\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:51:30.399778 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:30.399701    4962 generic.go:356] "Generic (PLEG): container finished" podID="fc110e07-7ec6-4455-9437-5b654b6cae44" containerID="5e68e3b2233253355cda330a1366d9bfa4f1965396b9362b9da19691b7a8aa97" exitCode=0
Oct 29 21:51:30.399778 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:30.399755    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"
Oct 29 21:51:30.400185 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:30.399783    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc" event={"ID":"fc110e07-7ec6-4455-9437-5b654b6cae44","Type":"ContainerDied","Data":"5e68e3b2233253355cda330a1366d9bfa4f1965396b9362b9da19691b7a8aa97"}
Oct 29 21:51:30.400185 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:30.399848    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc" event={"ID":"fc110e07-7ec6-4455-9437-5b654b6cae44","Type":"ContainerDied","Data":"5404f6e6de1d8660055c0920e674166b9989f3aa746439b0ce480bd4a92642a7"}
Oct 29 21:51:30.400185 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:30.399869    4962 scope.go:122] "RemoveContainer" containerID="5e68e3b2233253355cda330a1366d9bfa4f1965396b9362b9da19691b7a8aa97"
Oct 29 21:51:30.409006 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:30.408991    4962 scope.go:122] "RemoveContainer" containerID="5e68e3b2233253355cda330a1366d9bfa4f1965396b9362b9da19691b7a8aa97"
Oct 29 21:51:30.409258 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:51:30.409243    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e68e3b2233253355cda330a1366d9bfa4f1965396b9362b9da19691b7a8aa97\": container with ID starting with 5e68e3b2233253355cda330a1366d9bfa4f1965396b9362b9da19691b7a8aa97 not found: ID does not exist" containerID="5e68e3b2233253355cda330a1366d9bfa4f1965396b9362b9da19691b7a8aa97"
Oct 29 21:51:30.409325 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:30.409266    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e68e3b2233253355cda330a1366d9bfa4f1965396b9362b9da19691b7a8aa97"} err="failed to get container status \"5e68e3b2233253355cda330a1366d9bfa4f1965396b9362b9da19691b7a8aa97\": rpc error: code = NotFound desc = could not find container \"5e68e3b2233253355cda330a1366d9bfa4f1965396b9362b9da19691b7a8aa97\": container with ID starting with 5e68e3b2233253355cda330a1366d9bfa4f1965396b9362b9da19691b7a8aa97 not found: ID does not exist"
Oct 29 21:51:30.452349 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:30.452325    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"]
Oct 29 21:51:30.463931 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:30.463910    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xpzqc"]
Oct 29 21:51:31.056800 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:31.056766    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc110e07-7ec6-4455-9437-5b654b6cae44" path="/var/lib/kubelet/pods/fc110e07-7ec6-4455-9437-5b654b6cae44/volumes"
Oct 29 21:51:39.717941 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:51:39.717865    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:51:55.283783 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.283751    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79f9484d98-5snsz"]
Oct 29 21:51:55.284249 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.284065    4962 cpu_manager.go:414] "containerMap: removing container" podUID="fc110e07-7ec6-4455-9437-5b654b6cae44" containerName="discovery"
Oct 29 21:51:55.284249 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.284076    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc110e07-7ec6-4455-9437-5b654b6cae44" containerName="discovery"
Oct 29 21:51:55.284249 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.284146    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc110e07-7ec6-4455-9437-5b654b6cae44" containerName="discovery"
Oct 29 21:51:55.287253 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.287236    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79f9484d98-5snsz"
Oct 29 21:51:55.315121 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.315090    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 21:51:55.315238 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.315145    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 29 21:51:55.315238 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.315152    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 21:51:55.315366 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.315257    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 29 21:51:55.316060 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.316039    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2qkjk\""
Oct 29 21:51:55.328139 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.328112    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79f9484d98-5snsz"]
Oct 29 21:51:55.383327 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.383308    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b6c7990-30fa-4d71-bb45-54986a6fadf1-openshift-service-ca\") pod \"maas-api-79f9484d98-5snsz\" (UID: \"5b6c7990-30fa-4d71-bb45-54986a6fadf1\") " pod="odh-ai-gateway-infra/maas-api-79f9484d98-5snsz"
Oct 29 21:51:55.383419 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.383341    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b6c7990-30fa-4d71-bb45-54986a6fadf1-metrics-certs\") pod \"maas-api-79f9484d98-5snsz\" (UID: \"5b6c7990-30fa-4d71-bb45-54986a6fadf1\") " pod="odh-ai-gateway-infra/maas-api-79f9484d98-5snsz"
Oct 29 21:51:55.383463 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.383453    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b6c7990-30fa-4d71-bb45-54986a6fadf1-maas-api-tls\") pod \"maas-api-79f9484d98-5snsz\" (UID: \"5b6c7990-30fa-4d71-bb45-54986a6fadf1\") " pod="odh-ai-gateway-infra/maas-api-79f9484d98-5snsz"
Oct 29 21:51:55.383497 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.383479    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppkbz\" (UniqueName: \"kubernetes.io/projected/5b6c7990-30fa-4d71-bb45-54986a6fadf1-kube-api-access-ppkbz\") pod \"maas-api-79f9484d98-5snsz\" (UID: \"5b6c7990-30fa-4d71-bb45-54986a6fadf1\") " pod="odh-ai-gateway-infra/maas-api-79f9484d98-5snsz"
Oct 29 21:51:55.484346 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.484324    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b6c7990-30fa-4d71-bb45-54986a6fadf1-metrics-certs\") pod \"maas-api-79f9484d98-5snsz\" (UID: \"5b6c7990-30fa-4d71-bb45-54986a6fadf1\") " pod="odh-ai-gateway-infra/maas-api-79f9484d98-5snsz"
Oct 29 21:51:55.484426 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.484379    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b6c7990-30fa-4d71-bb45-54986a6fadf1-maas-api-tls\") pod \"maas-api-79f9484d98-5snsz\" (UID: \"5b6c7990-30fa-4d71-bb45-54986a6fadf1\") " pod="odh-ai-gateway-infra/maas-api-79f9484d98-5snsz"
Oct 29 21:51:55.484426 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.484404    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppkbz\" (UniqueName: \"kubernetes.io/projected/5b6c7990-30fa-4d71-bb45-54986a6fadf1-kube-api-access-ppkbz\") pod \"maas-api-79f9484d98-5snsz\" (UID: \"5b6c7990-30fa-4d71-bb45-54986a6fadf1\") " pod="odh-ai-gateway-infra/maas-api-79f9484d98-5snsz"
Oct 29 21:51:55.484426 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.484424    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b6c7990-30fa-4d71-bb45-54986a6fadf1-openshift-service-ca\") pod \"maas-api-79f9484d98-5snsz\" (UID: \"5b6c7990-30fa-4d71-bb45-54986a6fadf1\") " pod="odh-ai-gateway-infra/maas-api-79f9484d98-5snsz"
Oct 29 21:51:55.485066 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.485047    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b6c7990-30fa-4d71-bb45-54986a6fadf1-openshift-service-ca\") pod \"maas-api-79f9484d98-5snsz\" (UID: \"5b6c7990-30fa-4d71-bb45-54986a6fadf1\") " pod="odh-ai-gateway-infra/maas-api-79f9484d98-5snsz"
Oct 29 21:51:55.486769 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.486751    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b6c7990-30fa-4d71-bb45-54986a6fadf1-metrics-certs\") pod \"maas-api-79f9484d98-5snsz\" (UID: \"5b6c7990-30fa-4d71-bb45-54986a6fadf1\") " pod="odh-ai-gateway-infra/maas-api-79f9484d98-5snsz"
Oct 29 21:51:55.486863 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.486825    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b6c7990-30fa-4d71-bb45-54986a6fadf1-maas-api-tls\") pod \"maas-api-79f9484d98-5snsz\" (UID: \"5b6c7990-30fa-4d71-bb45-54986a6fadf1\") " pod="odh-ai-gateway-infra/maas-api-79f9484d98-5snsz"
Oct 29 21:51:55.508175 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.508146    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppkbz\" (UniqueName: \"kubernetes.io/projected/5b6c7990-30fa-4d71-bb45-54986a6fadf1-kube-api-access-ppkbz\") pod \"maas-api-79f9484d98-5snsz\" (UID: \"5b6c7990-30fa-4d71-bb45-54986a6fadf1\") " pod="odh-ai-gateway-infra/maas-api-79f9484d98-5snsz"
Oct 29 21:51:55.599360 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.599280    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79f9484d98-5snsz"
Oct 29 21:51:55.900033 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.899953    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79f9484d98-5snsz"]
Oct 29 21:51:55.902931 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:51:55.902904    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b6c7990_30fa_4d71_bb45_54986a6fadf1.slice/crio-2cbf819be2374370ad9b5733ab3045d1933b94ded2ebccb7ea4ad89e0fe61a54 WatchSource:0}: Error finding container 2cbf819be2374370ad9b5733ab3045d1933b94ded2ebccb7ea4ad89e0fe61a54: Status 404 returned error can't find the container with id 2cbf819be2374370ad9b5733ab3045d1933b94ded2ebccb7ea4ad89e0fe61a54
Oct 29 21:51:55.904237 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:55.904218    4962 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 21:51:56.507403 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:56.507367    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79f9484d98-5snsz" event={"ID":"5b6c7990-30fa-4d71-bb45-54986a6fadf1","Type":"ContainerStarted","Data":"2cbf819be2374370ad9b5733ab3045d1933b94ded2ebccb7ea4ad89e0fe61a54"}
Oct 29 21:51:58.518697 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:58.518666    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79f9484d98-5snsz" event={"ID":"5b6c7990-30fa-4d71-bb45-54986a6fadf1","Type":"ContainerStarted","Data":"45576b427397ebac292358927a5794545255ecff225b04867cefbf2a800781dc"}
Oct 29 21:51:58.519166 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:58.518783    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79f9484d98-5snsz"
Oct 29 21:51:58.565788 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:51:58.562582    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79f9484d98-5snsz" podStartSLOduration=1.368145713 podStartE2EDuration="3.562566802s" totalImagesPullingTime="2.194421089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:51:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:51:58.561274942 +0000 UTC m=+339.634231906" watchObservedRunningTime="2026-10-29 21:51:58.562566802 +0000 UTC m=+339.635523768"
Oct 29 21:52:04.528679 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:04.528650    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79f9484d98-5snsz"
Oct 29 21:52:09.718931 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:52:09.718898    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:52:14.610678 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:14.610641    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-79f9484d98-5snsz"]
Oct 29 21:52:14.611081 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:14.610853    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-79f9484d98-5snsz" podUID="5b6c7990-30fa-4d71-bb45-54986a6fadf1" containerName="maas-api" containerID="cri-o://45576b427397ebac292358927a5794545255ecff225b04867cefbf2a800781dc" gracePeriod=30
Oct 29 21:52:14.851060 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:14.851036    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79f9484d98-5snsz"
Oct 29 21:52:14.930035 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:14.930009    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b6c7990-30fa-4d71-bb45-54986a6fadf1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b6c7990-30fa-4d71-bb45-54986a6fadf1-maas-api-tls\") pod \"5b6c7990-30fa-4d71-bb45-54986a6fadf1\" (UID: \"5b6c7990-30fa-4d71-bb45-54986a6fadf1\") "
Oct 29 21:52:14.930206 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:14.930054    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b6c7990-30fa-4d71-bb45-54986a6fadf1-kube-api-access-ppkbz\" (UniqueName: \"kubernetes.io/projected/5b6c7990-30fa-4d71-bb45-54986a6fadf1-kube-api-access-ppkbz\") pod \"5b6c7990-30fa-4d71-bb45-54986a6fadf1\" (UID: \"5b6c7990-30fa-4d71-bb45-54986a6fadf1\") "
Oct 29 21:52:14.930206 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:14.930100    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b6c7990-30fa-4d71-bb45-54986a6fadf1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b6c7990-30fa-4d71-bb45-54986a6fadf1-metrics-certs\") pod \"5b6c7990-30fa-4d71-bb45-54986a6fadf1\" (UID: \"5b6c7990-30fa-4d71-bb45-54986a6fadf1\") "
Oct 29 21:52:14.930206 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:14.930149    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b6c7990-30fa-4d71-bb45-54986a6fadf1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b6c7990-30fa-4d71-bb45-54986a6fadf1-openshift-service-ca\") pod \"5b6c7990-30fa-4d71-bb45-54986a6fadf1\" (UID: \"5b6c7990-30fa-4d71-bb45-54986a6fadf1\") "
Oct 29 21:52:14.930591 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:14.930554    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b6c7990-30fa-4d71-bb45-54986a6fadf1-openshift-service-ca" pod "5b6c7990-30fa-4d71-bb45-54986a6fadf1" (UID: "5b6c7990-30fa-4d71-bb45-54986a6fadf1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:52:14.932265 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:14.932234    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b6c7990-30fa-4d71-bb45-54986a6fadf1-metrics-certs" pod "5b6c7990-30fa-4d71-bb45-54986a6fadf1" (UID: "5b6c7990-30fa-4d71-bb45-54986a6fadf1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:52:14.932397 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:14.932314    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b6c7990-30fa-4d71-bb45-54986a6fadf1-maas-api-tls" pod "5b6c7990-30fa-4d71-bb45-54986a6fadf1" (UID: "5b6c7990-30fa-4d71-bb45-54986a6fadf1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:52:14.932397 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:14.932353    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6c7990-30fa-4d71-bb45-54986a6fadf1-kube-api-access-ppkbz" pod "5b6c7990-30fa-4d71-bb45-54986a6fadf1" (UID: "5b6c7990-30fa-4d71-bb45-54986a6fadf1"). InnerVolumeSpecName "kube-api-access-ppkbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:52:15.031230 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:15.031205    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppkbz\" (UniqueName: \"kubernetes.io/projected/5b6c7990-30fa-4d71-bb45-54986a6fadf1-kube-api-access-ppkbz\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:52:15.031230 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:15.031230    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b6c7990-30fa-4d71-bb45-54986a6fadf1-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:52:15.031380 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:15.031239    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b6c7990-30fa-4d71-bb45-54986a6fadf1-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:52:15.031380 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:15.031248    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b6c7990-30fa-4d71-bb45-54986a6fadf1-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:52:15.588993 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:15.588957    4962 generic.go:356] "Generic (PLEG): container finished" podID="5b6c7990-30fa-4d71-bb45-54986a6fadf1" containerID="45576b427397ebac292358927a5794545255ecff225b04867cefbf2a800781dc" exitCode=0
Oct 29 21:52:15.589156 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:15.589010    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79f9484d98-5snsz"
Oct 29 21:52:15.589156 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:15.589024    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79f9484d98-5snsz" event={"ID":"5b6c7990-30fa-4d71-bb45-54986a6fadf1","Type":"ContainerDied","Data":"45576b427397ebac292358927a5794545255ecff225b04867cefbf2a800781dc"}
Oct 29 21:52:15.589156 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:15.589065    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79f9484d98-5snsz" event={"ID":"5b6c7990-30fa-4d71-bb45-54986a6fadf1","Type":"ContainerDied","Data":"2cbf819be2374370ad9b5733ab3045d1933b94ded2ebccb7ea4ad89e0fe61a54"}
Oct 29 21:52:15.589156 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:15.589088    4962 scope.go:122] "RemoveContainer" containerID="45576b427397ebac292358927a5794545255ecff225b04867cefbf2a800781dc"
Oct 29 21:52:15.597685 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:15.597666    4962 scope.go:122] "RemoveContainer" containerID="45576b427397ebac292358927a5794545255ecff225b04867cefbf2a800781dc"
Oct 29 21:52:15.597954 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:52:15.597925    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45576b427397ebac292358927a5794545255ecff225b04867cefbf2a800781dc\": container with ID starting with 45576b427397ebac292358927a5794545255ecff225b04867cefbf2a800781dc not found: ID does not exist" containerID="45576b427397ebac292358927a5794545255ecff225b04867cefbf2a800781dc"
Oct 29 21:52:15.598047 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:15.597963    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45576b427397ebac292358927a5794545255ecff225b04867cefbf2a800781dc"} err="failed to get container status \"45576b427397ebac292358927a5794545255ecff225b04867cefbf2a800781dc\": rpc error: code = NotFound desc = could not find container \"45576b427397ebac292358927a5794545255ecff225b04867cefbf2a800781dc\": container with ID starting with 45576b427397ebac292358927a5794545255ecff225b04867cefbf2a800781dc not found: ID does not exist"
Oct 29 21:52:15.643858 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:15.643832    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-79f9484d98-5snsz"]
Oct 29 21:52:15.655246 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:15.655219    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-79f9484d98-5snsz"]
Oct 29 21:52:17.055804 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:17.055771    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b6c7990-30fa-4d71-bb45-54986a6fadf1" path="/var/lib/kubelet/pods/5b6c7990-30fa-4d71-bb45-54986a6fadf1/volumes"
Oct 29 21:52:39.717708 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:52:39.717673    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:52:42.505830 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.505794    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"]
Oct 29 21:52:42.506206 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.506131    4962 cpu_manager.go:414] "containerMap: removing container" podUID="5b6c7990-30fa-4d71-bb45-54986a6fadf1" containerName="maas-api"
Oct 29 21:52:42.506206 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.506146    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b6c7990-30fa-4d71-bb45-54986a6fadf1" containerName="maas-api"
Oct 29 21:52:42.506280 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.506215    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b6c7990-30fa-4d71-bb45-54986a6fadf1" containerName="maas-api"
Oct 29 21:52:42.509097 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.509077    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.527317 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.527282    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-b4c9t\""
Oct 29 21:52:42.542867 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.542815    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"]
Oct 29 21:52:42.634166 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.634133    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92fb3ec-6127-46dd-a906-e84aec50ccae-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.634346 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.634178    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92fb3ec-6127-46dd-a906-e84aec50ccae-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.634346 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.634252    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92fb3ec-6127-46dd-a906-e84aec50ccae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.634456 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.634378    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92fb3ec-6127-46dd-a906-e84aec50ccae-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.634456 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.634419    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92fb3ec-6127-46dd-a906-e84aec50ccae-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.634518 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.634467    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92fb3ec-6127-46dd-a906-e84aec50ccae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.634518 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.634497    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92fb3ec-6127-46dd-a906-e84aec50ccae-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.634518 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.634514    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpff2\" (UniqueName: \"kubernetes.io/projected/a92fb3ec-6127-46dd-a906-e84aec50ccae-kube-api-access-qpff2\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.634616 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.634536    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92fb3ec-6127-46dd-a906-e84aec50ccae-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.735075 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.735052    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92fb3ec-6127-46dd-a906-e84aec50ccae-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.735215 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.735094    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92fb3ec-6127-46dd-a906-e84aec50ccae-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.735215 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.735122    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92fb3ec-6127-46dd-a906-e84aec50ccae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.735215 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.735147    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92fb3ec-6127-46dd-a906-e84aec50ccae-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.735215 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.735172    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpff2\" (UniqueName: \"kubernetes.io/projected/a92fb3ec-6127-46dd-a906-e84aec50ccae-kube-api-access-qpff2\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.735215 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.735199    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92fb3ec-6127-46dd-a906-e84aec50ccae-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.735507 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.735238    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92fb3ec-6127-46dd-a906-e84aec50ccae-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.735507 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.735278    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92fb3ec-6127-46dd-a906-e84aec50ccae-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.735507 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.735328    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92fb3ec-6127-46dd-a906-e84aec50ccae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.735657 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.735515    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92fb3ec-6127-46dd-a906-e84aec50ccae-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.735657 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.735623    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92fb3ec-6127-46dd-a906-e84aec50ccae-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.735771 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.735702    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92fb3ec-6127-46dd-a906-e84aec50ccae-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.735771 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.735722    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92fb3ec-6127-46dd-a906-e84aec50ccae-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.736030 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.736006    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92fb3ec-6127-46dd-a906-e84aec50ccae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.737535 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.737515    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92fb3ec-6127-46dd-a906-e84aec50ccae-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.737713 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.737695    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92fb3ec-6127-46dd-a906-e84aec50ccae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.758194 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.758137    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92fb3ec-6127-46dd-a906-e84aec50ccae-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.758194 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.758162    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpff2\" (UniqueName: \"kubernetes.io/projected/a92fb3ec-6127-46dd-a906-e84aec50ccae-kube-api-access-qpff2\") pod \"maas-default-gateway-openshift-default-769df7846c-j6f7m\" (UID: \"a92fb3ec-6127-46dd-a906-e84aec50ccae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:42.822331 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:42.822309    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:43.145174 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:43.145145    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"]
Oct 29 21:52:43.146849 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:52:43.146824    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda92fb3ec_6127_46dd_a906_e84aec50ccae.slice/crio-764f07b3636ffceaee052d03345227d3af7bad03606627c885466259a80be8c3 WatchSource:0}: Error finding container 764f07b3636ffceaee052d03345227d3af7bad03606627c885466259a80be8c3: Status 404 returned error can't find the container with id 764f07b3636ffceaee052d03345227d3af7bad03606627c885466259a80be8c3
Oct 29 21:52:43.705918 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:43.705885    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m" event={"ID":"a92fb3ec-6127-46dd-a906-e84aec50ccae","Type":"ContainerStarted","Data":"764f07b3636ffceaee052d03345227d3af7bad03606627c885466259a80be8c3"}
Oct 29 21:52:45.670760 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:45.670723    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:52:45.671185 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:45.670810    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:52:45.671185 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:45.670880    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:52:46.719817 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:46.719771    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m" event={"ID":"a92fb3ec-6127-46dd-a906-e84aec50ccae","Type":"ContainerStarted","Data":"5ae089574077d2658bd9bbfd43131e19baa0c4810aef2ad41a41de022d4fe051"}
Oct 29 21:52:46.787947 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:46.787881    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m" podStartSLOduration=2.266721815 podStartE2EDuration="4.787863604s" totalImagesPullingTime="2.521141789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:46.778678996 +0000 UTC m=+387.851635961" watchObservedRunningTime="2026-10-29 21:52:46.787863604 +0000 UTC m=+387.860820549"
Oct 29 21:52:46.822495 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:46.822458    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:46.824124 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:46.824099    4962 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-j6f7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 29 21:52:46.824255 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:46.824146    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m" podUID="a92fb3ec-6127-46dd-a906-e84aec50ccae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 29 21:52:47.823675 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:47.823640    4962 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-j6f7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 29 21:52:47.824068 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:47.823697    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m" podUID="a92fb3ec-6127-46dd-a906-e84aec50ccae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 29 21:52:48.823072 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:48.823035    4962 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-j6f7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 29 21:52:48.823236 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:48.823102    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m" podUID="a92fb3ec-6127-46dd-a906-e84aec50ccae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 29 21:52:49.823010 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:49.822973    4962 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-j6f7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 29 21:52:49.823449 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:49.823027    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m" podUID="a92fb3ec-6127-46dd-a906-e84aec50ccae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 29 21:52:50.823581 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:50.823541    4962 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-j6f7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Oct 29 21:52:50.824035 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:50.823600    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m" podUID="a92fb3ec-6127-46dd-a906-e84aec50ccae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Oct 29 21:52:52.567266 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:52.567234    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:52.567818 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:52.567792    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:52.568738 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:52.568710    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-j6f7m"
Oct 29 21:52:57.607823 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.607790    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"]
Oct 29 21:52:57.611365 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.611346    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.658043 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.658017    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"]
Oct 29 21:52:57.673417 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.673379    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.673417 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.673421    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.673575 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.673437    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.673575 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.673507    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.673575 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.673564    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dcff5f2-db63-4ae5-a935-7509c28c8867-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.673691 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.673602    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.673691 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.673660    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.673691 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.673679    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.673790 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.673719    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfgd9\" (UniqueName: \"kubernetes.io/projected/3dcff5f2-db63-4ae5-a935-7509c28c8867-kube-api-access-hfgd9\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.773999 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.773972    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.773999 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.774002    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.774168 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.774016    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.774168 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.774035    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.774168 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.774056    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dcff5f2-db63-4ae5-a935-7509c28c8867-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.774168 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.774097    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.774168 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.774127    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.774168 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.774153    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.774488 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.774218    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfgd9\" (UniqueName: \"kubernetes.io/projected/3dcff5f2-db63-4ae5-a935-7509c28c8867-kube-api-access-hfgd9\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.774539 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.774477    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.774616 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.774591    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.774669 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.774644    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.774724 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.774702    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.774998 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.774976    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dcff5f2-db63-4ae5-a935-7509c28c8867-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.776473 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.776455    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.776545 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.776525    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.797873 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.797847    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.805024 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.804997    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfgd9\" (UniqueName: \"kubernetes.io/projected/3dcff5f2-db63-4ae5-a935-7509c28c8867-kube-api-access-hfgd9\") pod \"maas-default-gateway-openshift-default-769df7846c-tz6f5\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:57.923627 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:57.923603    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:58.277980 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:58.277953    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"]
Oct 29 21:52:58.279605 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:52:58.279568    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcff5f2_db63_4ae5_a935_7509c28c8867.slice/crio-0525d2fd070b5c66eee7deed566b787b6af935f5633b3a5e62301dae8c921ca5 WatchSource:0}: Error finding container 0525d2fd070b5c66eee7deed566b787b6af935f5633b3a5e62301dae8c921ca5: Status 404 returned error can't find the container with id 0525d2fd070b5c66eee7deed566b787b6af935f5633b3a5e62301dae8c921ca5
Oct 29 21:52:58.281766 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:58.281736    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:52:58.281858 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:58.281797    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:52:58.281858 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:58.281826    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:52:58.768629 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:58.768583    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5" event={"ID":"3dcff5f2-db63-4ae5-a935-7509c28c8867","Type":"ContainerStarted","Data":"dcb6295765a76eb116d2a684331e402fcd7bf11bcfaad16fe2829f4f0dbb605d"}
Oct 29 21:52:58.769095 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:58.768640    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5" event={"ID":"3dcff5f2-db63-4ae5-a935-7509c28c8867","Type":"ContainerStarted","Data":"0525d2fd070b5c66eee7deed566b787b6af935f5633b3a5e62301dae8c921ca5"}
Oct 29 21:52:58.821436 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:58.821376    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5" podStartSLOduration=1.821356191 podStartE2EDuration="1.821356191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:52:58.820411336 +0000 UTC m=+399.893368340" watchObservedRunningTime="2026-10-29 21:52:58.821356191 +0000 UTC m=+399.894313156"
Oct 29 21:52:58.924569 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:58.924535    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:52:59.925817 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:59.925778    4962 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-tz6f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Oct 29 21:52:59.926196 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:52:59.925844    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5" podUID="3dcff5f2-db63-4ae5-a935-7509c28c8867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Oct 29 21:53:00.924540 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:00.924500    4962 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-tz6f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Oct 29 21:53:00.924708 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:00.924567    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5" podUID="3dcff5f2-db63-4ae5-a935-7509c28c8867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Oct 29 21:53:01.924412 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:01.924335    4962 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-tz6f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Oct 29 21:53:01.924412 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:01.924392    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5" podUID="3dcff5f2-db63-4ae5-a935-7509c28c8867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Oct 29 21:53:02.924766 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:02.924728    4962 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-tz6f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Oct 29 21:53:02.925183 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:02.924784    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5" podUID="3dcff5f2-db63-4ae5-a935-7509c28c8867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Oct 29 21:53:03.845748 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:03.845716    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-x26s8"]
Oct 29 21:53:03.849200 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:03.849180    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-x26s8"
Oct 29 21:53:03.866908 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:03.866884    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 29 21:53:03.894345 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:03.893652    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 29 21:53:03.894345 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:03.893727    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 29 21:53:03.894345 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:03.894024    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-htncd\""
Oct 29 21:53:03.894345 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:03.894071    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 29 21:53:03.901542 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:03.901475    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-x26s8"]
Oct 29 21:53:03.924520 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:03.924493    4962 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-tz6f5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body=
Oct 29 21:53:03.924657 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:03.924548    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5" podUID="3dcff5f2-db63-4ae5-a935-7509c28c8867" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused"
Oct 29 21:53:03.936012 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:03.935934    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjvzq\" (UniqueName: \"kubernetes.io/projected/7d041360-9f79-4c0c-8b53-4086db301662-kube-api-access-xjvzq\") pod \"ai-gateway-controller-7d85675f98-x26s8\" (UID: \"7d041360-9f79-4c0c-8b53-4086db301662\") " pod="opendatahub/ai-gateway-controller-7d85675f98-x26s8"
Oct 29 21:53:03.936501 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:03.936038    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d041360-9f79-4c0c-8b53-4086db301662-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-x26s8\" (UID: \"7d041360-9f79-4c0c-8b53-4086db301662\") " pod="opendatahub/ai-gateway-controller-7d85675f98-x26s8"
Oct 29 21:53:04.036756 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:04.036725    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjvzq\" (UniqueName: \"kubernetes.io/projected/7d041360-9f79-4c0c-8b53-4086db301662-kube-api-access-xjvzq\") pod \"ai-gateway-controller-7d85675f98-x26s8\" (UID: \"7d041360-9f79-4c0c-8b53-4086db301662\") " pod="opendatahub/ai-gateway-controller-7d85675f98-x26s8"
Oct 29 21:53:04.036756 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:04.036760    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d041360-9f79-4c0c-8b53-4086db301662-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-x26s8\" (UID: \"7d041360-9f79-4c0c-8b53-4086db301662\") " pod="opendatahub/ai-gateway-controller-7d85675f98-x26s8"
Oct 29 21:53:04.039326 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:04.039303    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d041360-9f79-4c0c-8b53-4086db301662-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-x26s8\" (UID: \"7d041360-9f79-4c0c-8b53-4086db301662\") " pod="opendatahub/ai-gateway-controller-7d85675f98-x26s8"
Oct 29 21:53:04.061629 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:04.061605    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjvzq\" (UniqueName: \"kubernetes.io/projected/7d041360-9f79-4c0c-8b53-4086db301662-kube-api-access-xjvzq\") pod \"ai-gateway-controller-7d85675f98-x26s8\" (UID: \"7d041360-9f79-4c0c-8b53-4086db301662\") " pod="opendatahub/ai-gateway-controller-7d85675f98-x26s8"
Oct 29 21:53:04.162091 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:04.162068    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-x26s8"
Oct 29 21:53:04.517252 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:04.517131    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-x26s8"]
Oct 29 21:53:04.519993 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:53:04.519966    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d041360_9f79_4c0c_8b53_4086db301662.slice/crio-8fca9358df677e76acfefdb544450c2b6f04381926a29c91fcc30b1887a6a5ae WatchSource:0}: Error finding container 8fca9358df677e76acfefdb544450c2b6f04381926a29c91fcc30b1887a6a5ae: Status 404 returned error can't find the container with id 8fca9358df677e76acfefdb544450c2b6f04381926a29c91fcc30b1887a6a5ae
Oct 29 21:53:04.813270 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:04.813185    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-x26s8" event={"ID":"7d041360-9f79-4c0c-8b53-4086db301662","Type":"ContainerStarted","Data":"8fca9358df677e76acfefdb544450c2b6f04381926a29c91fcc30b1887a6a5ae"}
Oct 29 21:53:05.387347 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:05.387312    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:53:05.387768 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:05.387719    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:53:05.389543 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:05.389519    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 21:53:06.824029 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:06.823990    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-x26s8" event={"ID":"7d041360-9f79-4c0c-8b53-4086db301662","Type":"ContainerStarted","Data":"f39648e4cef5c857398e52538550509515d27584f822843c45a6861d78e86782"}
Oct 29 21:53:06.824457 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:06.824317    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-x26s8"
Oct 29 21:53:08.417582 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:08.417528    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-x26s8" podStartSLOduration=3.997653512 podStartE2EDuration="5.417514116s" totalImagesPullingTime="1.419860604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:06.869227709 +0000 UTC m=+407.942184673" watchObservedRunningTime="2026-10-29 21:53:08.417514116 +0000 UTC m=+409.490471079"
Oct 29 21:53:08.418785 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:08.418759    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-trj5g"]
Oct 29 21:53:08.422250 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:08.422228    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-trj5g"
Oct 29 21:53:08.449529 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:08.449502    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 29 21:53:08.449529 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:08.449524    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kkgls\""
Oct 29 21:53:08.450857 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:08.450841    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-trj5g"]
Oct 29 21:53:08.469175 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:08.469150    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57lwj\" (UniqueName: \"kubernetes.io/projected/3973cd94-3216-4920-a993-0ea710831e20-kube-api-access-57lwj\") pod \"payload-pre-processing-7b98bb678c-trj5g\" (UID: \"3973cd94-3216-4920-a993-0ea710831e20\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-trj5g"
Oct 29 21:53:08.469489 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:08.469468    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3973cd94-3216-4920-a993-0ea710831e20-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-trj5g\" (UID: \"3973cd94-3216-4920-a993-0ea710831e20\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-trj5g"
Oct 29 21:53:08.497683 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:08.497647    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-d4fgq"]
Oct 29 21:53:08.501272 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:08.501242    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-d4fgq"
Oct 29 21:53:08.524680 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:08.524644    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-d4fgq"]
Oct 29 21:53:08.570103 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:08.570073    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slh24\" (UniqueName: \"kubernetes.io/projected/3472b532-36f0-4a0d-9d04-27ec9a9f32e6-kube-api-access-slh24\") pod \"payload-processing-5f78687d5b-d4fgq\" (UID: \"3472b532-36f0-4a0d-9d04-27ec9a9f32e6\") " pod="openshift-ingress/payload-processing-5f78687d5b-d4fgq"
Oct 29 21:53:08.570253 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:08.570122    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3472b532-36f0-4a0d-9d04-27ec9a9f32e6-plugins-config-volume\") pod \"payload-processing-5f78687d5b-d4fgq\" (UID: \"3472b532-36f0-4a0d-9d04-27ec9a9f32e6\") " pod="openshift-ingress/payload-processing-5f78687d5b-d4fgq"
Oct 29 21:53:08.570253 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:08.570224    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3973cd94-3216-4920-a993-0ea710831e20-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-trj5g\" (UID: \"3973cd94-3216-4920-a993-0ea710831e20\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-trj5g"
Oct 29 21:53:08.570371 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:08.570275    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57lwj\" (UniqueName: \"kubernetes.io/projected/3973cd94-3216-4920-a993-0ea710831e20-kube-api-access-57lwj\") pod \"payload-pre-processing-7b98bb678c-trj5g\" (UID: \"3973cd94-3216-4920-a993-0ea710831e20\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-trj5g"
Oct 29 21:53:08.570973 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:08.570952    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3973cd94-3216-4920-a993-0ea710831e20-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-trj5g\" (UID: \"3973cd94-3216-4920-a993-0ea710831e20\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-trj5g"
Oct 29 21:53:08.596435 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:08.596402    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57lwj\" (UniqueName: \"kubernetes.io/projected/3973cd94-3216-4920-a993-0ea710831e20-kube-api-access-57lwj\") pod \"payload-pre-processing-7b98bb678c-trj5g\" (UID: \"3973cd94-3216-4920-a993-0ea710831e20\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-trj5g"
Oct 29 21:53:08.670983 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:08.670919    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3472b532-36f0-4a0d-9d04-27ec9a9f32e6-plugins-config-volume\") pod \"payload-processing-5f78687d5b-d4fgq\" (UID: \"3472b532-36f0-4a0d-9d04-27ec9a9f32e6\") " pod="openshift-ingress/payload-processing-5f78687d5b-d4fgq"
Oct 29 21:53:08.671124 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:08.670985    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slh24\" (UniqueName: \"kubernetes.io/projected/3472b532-36f0-4a0d-9d04-27ec9a9f32e6-kube-api-access-slh24\") pod \"payload-processing-5f78687d5b-d4fgq\" (UID: \"3472b532-36f0-4a0d-9d04-27ec9a9f32e6\") " pod="openshift-ingress/payload-processing-5f78687d5b-d4fgq"
Oct 29 21:53:08.671622 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:08.671599    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3472b532-36f0-4a0d-9d04-27ec9a9f32e6-plugins-config-volume\") pod \"payload-processing-5f78687d5b-d4fgq\" (UID: \"3472b532-36f0-4a0d-9d04-27ec9a9f32e6\") " pod="openshift-ingress/payload-processing-5f78687d5b-d4fgq"
Oct 29 21:53:08.694527 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:08.694505    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slh24\" (UniqueName: \"kubernetes.io/projected/3472b532-36f0-4a0d-9d04-27ec9a9f32e6-kube-api-access-slh24\") pod \"payload-processing-5f78687d5b-d4fgq\" (UID: \"3472b532-36f0-4a0d-9d04-27ec9a9f32e6\") " pod="openshift-ingress/payload-processing-5f78687d5b-d4fgq"
Oct 29 21:53:08.733918 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:08.733895    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-trj5g"
Oct 29 21:53:08.814381 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:08.814345    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-d4fgq"
Oct 29 21:53:09.028839 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:09.028651    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-trj5g"]
Oct 29 21:53:09.031370 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:53:09.031344    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3973cd94_3216_4920_a993_0ea710831e20.slice/crio-33a5889c0bcca2140380938a8539c43ce98be29f096a4df7a32fd4a405d5b180 WatchSource:0}: Error finding container 33a5889c0bcca2140380938a8539c43ce98be29f096a4df7a32fd4a405d5b180: Status 404 returned error can't find the container with id 33a5889c0bcca2140380938a8539c43ce98be29f096a4df7a32fd4a405d5b180
Oct 29 21:53:09.110777 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:09.110753    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-d4fgq"]
Oct 29 21:53:09.112958 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:53:09.112922    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3472b532_36f0_4a0d_9d04_27ec9a9f32e6.slice/crio-1a72a41f8b99e3833841019010fdadd73b82d4208c3dff110eb277cc5ad43764 WatchSource:0}: Error finding container 1a72a41f8b99e3833841019010fdadd73b82d4208c3dff110eb277cc5ad43764: Status 404 returned error can't find the container with id 1a72a41f8b99e3833841019010fdadd73b82d4208c3dff110eb277cc5ad43764
Oct 29 21:53:09.730544 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:53:09.730503    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:53:09.844969 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:09.844921    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-trj5g" event={"ID":"3973cd94-3216-4920-a993-0ea710831e20","Type":"ContainerStarted","Data":"33a5889c0bcca2140380938a8539c43ce98be29f096a4df7a32fd4a405d5b180"}
Oct 29 21:53:09.846525 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:09.846495    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-d4fgq" event={"ID":"3472b532-36f0-4a0d-9d04-27ec9a9f32e6","Type":"ContainerStarted","Data":"1a72a41f8b99e3833841019010fdadd73b82d4208c3dff110eb277cc5ad43764"}
Oct 29 21:53:11.856518 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:11.856479    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-trj5g" event={"ID":"3973cd94-3216-4920-a993-0ea710831e20","Type":"ContainerStarted","Data":"0cb8a8dda70917ee8afd76574720080cf019fa02c359f6d658a5fa95faba3a5e"}
Oct 29 21:53:11.856964 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:11.856637    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-trj5g"
Oct 29 21:53:11.858067 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:11.858045    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-d4fgq" event={"ID":"3472b532-36f0-4a0d-9d04-27ec9a9f32e6","Type":"ContainerStarted","Data":"0275810c1e79f6d032b41cf0a832be87c3e0a0ce4df5b60f059a9c052e9fdddc"}
Oct 29 21:53:11.858185 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:11.858092    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-d4fgq"
Oct 29 21:53:11.899498 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:11.899456    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-trj5g" podStartSLOduration=1.6609112449999999 podStartE2EDuration="3.899443979s" totalImagesPullingTime="2.238532734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:11.898382545 +0000 UTC m=+412.971339520" watchObservedRunningTime="2026-10-29 21:53:11.899443979 +0000 UTC m=+412.972400983"
Oct 29 21:53:11.940475 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:11.940436    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-d4fgq" podStartSLOduration=1.774470754 podStartE2EDuration="3.940424439s" totalImagesPullingTime="2.165953685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:11.939525864 +0000 UTC m=+413.012482828" watchObservedRunningTime="2026-10-29 21:53:11.940424439 +0000 UTC m=+413.013381449"
Oct 29 21:53:18.838069 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:18.838034    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-x26s8"
Oct 29 21:53:19.124793 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:19.124721    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-767df45bdd-6djqs"]
Oct 29 21:53:19.134130 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:19.134088    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-767df45bdd-6djqs"
Oct 29 21:53:19.148725 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:19.148702    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e70e0c-c896-4641-b5d7-f97e8bf2ce79-metrics-certs\") pod \"ai-gateway-controller-767df45bdd-6djqs\" (UID: \"68e70e0c-c896-4641-b5d7-f97e8bf2ce79\") " pod="opendatahub/ai-gateway-controller-767df45bdd-6djqs"
Oct 29 21:53:19.149079 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:19.149053    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4kkj\" (UniqueName: \"kubernetes.io/projected/68e70e0c-c896-4641-b5d7-f97e8bf2ce79-kube-api-access-c4kkj\") pod \"ai-gateway-controller-767df45bdd-6djqs\" (UID: \"68e70e0c-c896-4641-b5d7-f97e8bf2ce79\") " pod="opendatahub/ai-gateway-controller-767df45bdd-6djqs"
Oct 29 21:53:19.150607 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:19.150585    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-767df45bdd-6djqs"]
Oct 29 21:53:19.249417 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:19.249390    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e70e0c-c896-4641-b5d7-f97e8bf2ce79-metrics-certs\") pod \"ai-gateway-controller-767df45bdd-6djqs\" (UID: \"68e70e0c-c896-4641-b5d7-f97e8bf2ce79\") " pod="opendatahub/ai-gateway-controller-767df45bdd-6djqs"
Oct 29 21:53:19.249570 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:19.249501    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4kkj\" (UniqueName: \"kubernetes.io/projected/68e70e0c-c896-4641-b5d7-f97e8bf2ce79-kube-api-access-c4kkj\") pod \"ai-gateway-controller-767df45bdd-6djqs\" (UID: \"68e70e0c-c896-4641-b5d7-f97e8bf2ce79\") " pod="opendatahub/ai-gateway-controller-767df45bdd-6djqs"
Oct 29 21:53:19.252132 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:19.252111    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68e70e0c-c896-4641-b5d7-f97e8bf2ce79-metrics-certs\") pod \"ai-gateway-controller-767df45bdd-6djqs\" (UID: \"68e70e0c-c896-4641-b5d7-f97e8bf2ce79\") " pod="opendatahub/ai-gateway-controller-767df45bdd-6djqs"
Oct 29 21:53:19.272890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:19.272859    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4kkj\" (UniqueName: \"kubernetes.io/projected/68e70e0c-c896-4641-b5d7-f97e8bf2ce79-kube-api-access-c4kkj\") pod \"ai-gateway-controller-767df45bdd-6djqs\" (UID: \"68e70e0c-c896-4641-b5d7-f97e8bf2ce79\") " pod="opendatahub/ai-gateway-controller-767df45bdd-6djqs"
Oct 29 21:53:19.447333 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:19.447313    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-767df45bdd-6djqs"
Oct 29 21:53:19.744326 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:19.744302    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-767df45bdd-6djqs"]
Oct 29 21:53:19.746912 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:53:19.746868    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e70e0c_c896_4641_b5d7_f97e8bf2ce79.slice/crio-e33e0ac1407a118f24aa49400ae05539fa27cbe687a9af1ea29e43448df650f4 WatchSource:0}: Error finding container e33e0ac1407a118f24aa49400ae05539fa27cbe687a9af1ea29e43448df650f4: Status 404 returned error can't find the container with id e33e0ac1407a118f24aa49400ae05539fa27cbe687a9af1ea29e43448df650f4
Oct 29 21:53:19.893482 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:19.893447    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-767df45bdd-6djqs" event={"ID":"68e70e0c-c896-4641-b5d7-f97e8bf2ce79","Type":"ContainerStarted","Data":"e33e0ac1407a118f24aa49400ae05539fa27cbe687a9af1ea29e43448df650f4"}
Oct 29 21:53:20.900008 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:20.899963    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-767df45bdd-6djqs" event={"ID":"68e70e0c-c896-4641-b5d7-f97e8bf2ce79","Type":"ContainerStarted","Data":"af5df6f65b105900cbf29285202d61ba625621dcf003910f8dcce670d55e1181"}
Oct 29 21:53:20.900417 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:20.900159    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-767df45bdd-6djqs"
Oct 29 21:53:20.942509 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:20.942468    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-767df45bdd-6djqs" podStartSLOduration=1.5255439640000001 podStartE2EDuration="1.942455586s" totalImagesPullingTime="416.911622ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:20.941022579 +0000 UTC m=+422.013979629" watchObservedRunningTime="2026-10-29 21:53:20.942455586 +0000 UTC m=+422.015412549"
Oct 29 21:53:22.865258 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:22.865228    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-d4fgq"
Oct 29 21:53:22.865697 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:22.865318    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-trj5g"
Oct 29 21:53:27.690885 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.690845    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"]
Oct 29 21:53:27.694706 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.694682    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.723062 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.723038    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"]
Oct 29 21:53:27.819382 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.819358    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.819512 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.819388    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.819512 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.819409    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.819512 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.819427    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.819610 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.819513    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029c7ea5-bf31-4b22-96b8-41a693287efe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.819610 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.819572    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9r24\" (UniqueName: \"kubernetes.io/projected/029c7ea5-bf31-4b22-96b8-41a693287efe-kube-api-access-r9r24\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.819673 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.819616    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.819705 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.819684    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.819741 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.819703    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.920392 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.920368    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.920514 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.920396    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.920514 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.920411    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.920514 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.920428    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.920514 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.920455    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029c7ea5-bf31-4b22-96b8-41a693287efe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.920514 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.920479    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9r24\" (UniqueName: \"kubernetes.io/projected/029c7ea5-bf31-4b22-96b8-41a693287efe-kube-api-access-r9r24\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.920757 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.920534    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.920757 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.920578    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.920757 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.920604    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.920934 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.920914    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.921014 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.920964    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.921014 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.920998    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.921111 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.921086    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.921451 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.921424    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029c7ea5-bf31-4b22-96b8-41a693287efe-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.922964 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.922945    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.923433 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.923413    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.943565 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.943514    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:27.943696 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:27.943675    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9r24\" (UniqueName: \"kubernetes.io/projected/029c7ea5-bf31-4b22-96b8-41a693287efe-kube-api-access-r9r24\") pod \"maas-default-gateway-openshift-default-769df7846c-zz6dk\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:28.008990 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:28.008965    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:28.323611 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:28.323542    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"]
Oct 29 21:53:28.326453 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:53:28.326425    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029c7ea5_bf31_4b22_96b8_41a693287efe.slice/crio-ccefe3046589b53dd91037b132f41e76cef5be5a2e44d15f9d179d6faf8dbaad WatchSource:0}: Error finding container ccefe3046589b53dd91037b132f41e76cef5be5a2e44d15f9d179d6faf8dbaad: Status 404 returned error can't find the container with id ccefe3046589b53dd91037b132f41e76cef5be5a2e44d15f9d179d6faf8dbaad
Oct 29 21:53:28.328612 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:28.328580    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:53:28.328697 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:28.328662    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:53:28.328765 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:28.328705    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:53:28.938354 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:28.938320    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk" event={"ID":"029c7ea5-bf31-4b22-96b8-41a693287efe","Type":"ContainerStarted","Data":"e7db17f06ec1c34a7c7da47638cebdfa2c9c6da89800d405626cbb9c11101551"}
Oct 29 21:53:28.938799 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:28.938379    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk" event={"ID":"029c7ea5-bf31-4b22-96b8-41a693287efe","Type":"ContainerStarted","Data":"ccefe3046589b53dd91037b132f41e76cef5be5a2e44d15f9d179d6faf8dbaad"}
Oct 29 21:53:28.987170 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:28.987110    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk" podStartSLOduration=1.987090016 podStartE2EDuration="1.987090016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:28.986937727 +0000 UTC m=+430.059894692" watchObservedRunningTime="2026-10-29 21:53:28.987090016 +0000 UTC m=+430.060046986"
Oct 29 21:53:29.009940 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:29.009903    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:29.011545 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:29.011519    4962 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-zz6dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 29 21:53:29.011673 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:29.011566    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk" podUID="029c7ea5-bf31-4b22-96b8-41a693287efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 29 21:53:30.010213 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:30.010175    4962 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-zz6dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 29 21:53:30.010630 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:30.010232    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk" podUID="029c7ea5-bf31-4b22-96b8-41a693287efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 29 21:53:31.009797 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:31.009761    4962 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-zz6dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 29 21:53:31.009967 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:31.009815    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk" podUID="029c7ea5-bf31-4b22-96b8-41a693287efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 29 21:53:31.906164 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:31.906135    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-767df45bdd-6djqs"
Oct 29 21:53:31.992848 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:31.992799    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-x26s8"]
Oct 29 21:53:31.998587 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:31.998532    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-x26s8" podUID="7d041360-9f79-4c0c-8b53-4086db301662" containerName="manager" containerID="cri-o://f39648e4cef5c857398e52538550509515d27584f822843c45a6861d78e86782" gracePeriod=30
Oct 29 21:53:32.010279 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:32.010252    4962 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-zz6dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 29 21:53:32.010831 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:32.010803    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk" podUID="029c7ea5-bf31-4b22-96b8-41a693287efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 29 21:53:32.257118 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:32.257092    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-x26s8"
Oct 29 21:53:32.356513 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:32.356483    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d041360-9f79-4c0c-8b53-4086db301662-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d041360-9f79-4c0c-8b53-4086db301662-metrics-certs\") pod \"7d041360-9f79-4c0c-8b53-4086db301662\" (UID: \"7d041360-9f79-4c0c-8b53-4086db301662\") "
Oct 29 21:53:32.356685 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:32.356528    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d041360-9f79-4c0c-8b53-4086db301662-kube-api-access-xjvzq\" (UniqueName: \"kubernetes.io/projected/7d041360-9f79-4c0c-8b53-4086db301662-kube-api-access-xjvzq\") pod \"7d041360-9f79-4c0c-8b53-4086db301662\" (UID: \"7d041360-9f79-4c0c-8b53-4086db301662\") "
Oct 29 21:53:32.358676 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:32.358649    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d041360-9f79-4c0c-8b53-4086db301662-metrics-certs" pod "7d041360-9f79-4c0c-8b53-4086db301662" (UID: "7d041360-9f79-4c0c-8b53-4086db301662"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:53:32.358789 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:32.358696    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d041360-9f79-4c0c-8b53-4086db301662-kube-api-access-xjvzq" pod "7d041360-9f79-4c0c-8b53-4086db301662" (UID: "7d041360-9f79-4c0c-8b53-4086db301662"). InnerVolumeSpecName "kube-api-access-xjvzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:53:32.457328 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:32.457239    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjvzq\" (UniqueName: \"kubernetes.io/projected/7d041360-9f79-4c0c-8b53-4086db301662-kube-api-access-xjvzq\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:53:32.457328 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:32.457267    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d041360-9f79-4c0c-8b53-4086db301662-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:53:32.956157 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:32.956123    4962 generic.go:356] "Generic (PLEG): container finished" podID="7d041360-9f79-4c0c-8b53-4086db301662" containerID="f39648e4cef5c857398e52538550509515d27584f822843c45a6861d78e86782" exitCode=0
Oct 29 21:53:32.956660 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:32.956167    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-x26s8"
Oct 29 21:53:32.956660 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:32.956183    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-x26s8" event={"ID":"7d041360-9f79-4c0c-8b53-4086db301662","Type":"ContainerDied","Data":"f39648e4cef5c857398e52538550509515d27584f822843c45a6861d78e86782"}
Oct 29 21:53:32.956660 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:32.956247    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-x26s8" event={"ID":"7d041360-9f79-4c0c-8b53-4086db301662","Type":"ContainerDied","Data":"8fca9358df677e76acfefdb544450c2b6f04381926a29c91fcc30b1887a6a5ae"}
Oct 29 21:53:32.956660 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:32.956278    4962 scope.go:122] "RemoveContainer" containerID="f39648e4cef5c857398e52538550509515d27584f822843c45a6861d78e86782"
Oct 29 21:53:32.965236 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:32.965219    4962 scope.go:122] "RemoveContainer" containerID="f39648e4cef5c857398e52538550509515d27584f822843c45a6861d78e86782"
Oct 29 21:53:32.965511 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:53:32.965489    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f39648e4cef5c857398e52538550509515d27584f822843c45a6861d78e86782\": container with ID starting with f39648e4cef5c857398e52538550509515d27584f822843c45a6861d78e86782 not found: ID does not exist" containerID="f39648e4cef5c857398e52538550509515d27584f822843c45a6861d78e86782"
Oct 29 21:53:32.965581 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:32.965518    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39648e4cef5c857398e52538550509515d27584f822843c45a6861d78e86782"} err="failed to get container status \"f39648e4cef5c857398e52538550509515d27584f822843c45a6861d78e86782\": rpc error: code = NotFound desc = could not find container \"f39648e4cef5c857398e52538550509515d27584f822843c45a6861d78e86782\": container with ID starting with f39648e4cef5c857398e52538550509515d27584f822843c45a6861d78e86782 not found: ID does not exist"
Oct 29 21:53:33.009792 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:33.009764    4962 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-zz6dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 29 21:53:33.009917 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:33.009810    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk" podUID="029c7ea5-bf31-4b22-96b8-41a693287efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 29 21:53:33.033612 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:33.033578    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-x26s8"]
Oct 29 21:53:33.042344 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:33.042317    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-x26s8"]
Oct 29 21:53:33.055572 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:33.055549    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d041360-9f79-4c0c-8b53-4086db301662" path="/var/lib/kubelet/pods/7d041360-9f79-4c0c-8b53-4086db301662/volumes"
Oct 29 21:53:34.010207 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:34.010171    4962 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-zz6dk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Oct 29 21:53:34.010623 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:34.010234    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk" podUID="029c7ea5-bf31-4b22-96b8-41a693287efe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Oct 29 21:53:35.013735 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:35.013700    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:35.014117 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:35.013962    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:35.014623 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:35.014607    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 21:53:39.718380 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:53:39.718342    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:53:48.050266 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.050234    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"]
Oct 29 21:53:48.050658 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.050633    4962 cpu_manager.go:414] "containerMap: removing container" podUID="7d041360-9f79-4c0c-8b53-4086db301662" containerName="manager"
Oct 29 21:53:48.050658 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.050646    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d041360-9f79-4c0c-8b53-4086db301662" containerName="manager"
Oct 29 21:53:48.050730 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.050718    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d041360-9f79-4c0c-8b53-4086db301662" containerName="manager"
Oct 29 21:53:48.058390 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.058354    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 21:53:48.075898 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.075876    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 29 21:53:48.087454 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.087436    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 29 21:53:48.087722 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.087702    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5z45z\""
Oct 29 21:53:48.087810 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.087795    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 29 21:53:48.089137 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.089117    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"]
Oct 29 21:53:48.195727 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.195702    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70e1bafe-4b93-44d7-a179-44d5a82fd069-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fqldg\" (UID: \"70e1bafe-4b93-44d7-a179-44d5a82fd069\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 21:53:48.195842 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.195760    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swp4x\" (UniqueName: \"kubernetes.io/projected/70e1bafe-4b93-44d7-a179-44d5a82fd069-kube-api-access-swp4x\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fqldg\" (UID: \"70e1bafe-4b93-44d7-a179-44d5a82fd069\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 21:53:48.195883 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.195861    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70e1bafe-4b93-44d7-a179-44d5a82fd069-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fqldg\" (UID: \"70e1bafe-4b93-44d7-a179-44d5a82fd069\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 21:53:48.195922 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.195895    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70e1bafe-4b93-44d7-a179-44d5a82fd069-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fqldg\" (UID: \"70e1bafe-4b93-44d7-a179-44d5a82fd069\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 21:53:48.196003 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.195990    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70e1bafe-4b93-44d7-a179-44d5a82fd069-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fqldg\" (UID: \"70e1bafe-4b93-44d7-a179-44d5a82fd069\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 21:53:48.196038 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.196026    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70e1bafe-4b93-44d7-a179-44d5a82fd069-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fqldg\" (UID: \"70e1bafe-4b93-44d7-a179-44d5a82fd069\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 21:53:48.296649 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.296606    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swp4x\" (UniqueName: \"kubernetes.io/projected/70e1bafe-4b93-44d7-a179-44d5a82fd069-kube-api-access-swp4x\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fqldg\" (UID: \"70e1bafe-4b93-44d7-a179-44d5a82fd069\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 21:53:48.296834 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.296677    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70e1bafe-4b93-44d7-a179-44d5a82fd069-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fqldg\" (UID: \"70e1bafe-4b93-44d7-a179-44d5a82fd069\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 21:53:48.296834 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.296710    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70e1bafe-4b93-44d7-a179-44d5a82fd069-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fqldg\" (UID: \"70e1bafe-4b93-44d7-a179-44d5a82fd069\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 21:53:48.296963 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.296784    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70e1bafe-4b93-44d7-a179-44d5a82fd069-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fqldg\" (UID: \"70e1bafe-4b93-44d7-a179-44d5a82fd069\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 21:53:48.297191 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.297167    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70e1bafe-4b93-44d7-a179-44d5a82fd069-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fqldg\" (UID: \"70e1bafe-4b93-44d7-a179-44d5a82fd069\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 21:53:48.297480 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.297450    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70e1bafe-4b93-44d7-a179-44d5a82fd069-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fqldg\" (UID: \"70e1bafe-4b93-44d7-a179-44d5a82fd069\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 21:53:48.297480 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.297466    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70e1bafe-4b93-44d7-a179-44d5a82fd069-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fqldg\" (UID: \"70e1bafe-4b93-44d7-a179-44d5a82fd069\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 21:53:48.297654 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.297164    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70e1bafe-4b93-44d7-a179-44d5a82fd069-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fqldg\" (UID: \"70e1bafe-4b93-44d7-a179-44d5a82fd069\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 21:53:48.297763 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.297688    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70e1bafe-4b93-44d7-a179-44d5a82fd069-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fqldg\" (UID: \"70e1bafe-4b93-44d7-a179-44d5a82fd069\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 21:53:48.300661 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.300569    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70e1bafe-4b93-44d7-a179-44d5a82fd069-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fqldg\" (UID: \"70e1bafe-4b93-44d7-a179-44d5a82fd069\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 21:53:48.300870 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.300851    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70e1bafe-4b93-44d7-a179-44d5a82fd069-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fqldg\" (UID: \"70e1bafe-4b93-44d7-a179-44d5a82fd069\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 21:53:48.320258 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.320231    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swp4x\" (UniqueName: \"kubernetes.io/projected/70e1bafe-4b93-44d7-a179-44d5a82fd069-kube-api-access-swp4x\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-fqldg\" (UID: \"70e1bafe-4b93-44d7-a179-44d5a82fd069\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 21:53:48.372920 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.372877    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 21:53:48.687312 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:48.684777    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"]
Oct 29 21:53:48.719736 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:53:48.719687    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e1bafe_4b93_44d7_a179_44d5a82fd069.slice/crio-00a88b701ea6a1c162dd53bbeea592783d0016983738c5d3a6701b2a19835580 WatchSource:0}: Error finding container 00a88b701ea6a1c162dd53bbeea592783d0016983738c5d3a6701b2a19835580: Status 404 returned error can't find the container with id 00a88b701ea6a1c162dd53bbeea592783d0016983738c5d3a6701b2a19835580
Oct 29 21:53:49.031897 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.031818    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg" event={"ID":"70e1bafe-4b93-44d7-a179-44d5a82fd069","Type":"ContainerStarted","Data":"00a88b701ea6a1c162dd53bbeea592783d0016983738c5d3a6701b2a19835580"}
Oct 29 21:53:49.283907 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.283840    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"]
Oct 29 21:53:49.289555 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.289531    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 21:53:49.315627 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.315582    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 29 21:53:49.318037 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.318012    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"]
Oct 29 21:53:49.427032 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.427001    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a85f7d35-4bff-45d0-a392-2ecf056c2051-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xnbks\" (UID: \"a85f7d35-4bff-45d0-a392-2ecf056c2051\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 21:53:49.427032 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.427037    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrn2b\" (UniqueName: \"kubernetes.io/projected/a85f7d35-4bff-45d0-a392-2ecf056c2051-kube-api-access-wrn2b\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xnbks\" (UID: \"a85f7d35-4bff-45d0-a392-2ecf056c2051\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 21:53:49.427230 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.427140    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a85f7d35-4bff-45d0-a392-2ecf056c2051-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xnbks\" (UID: \"a85f7d35-4bff-45d0-a392-2ecf056c2051\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 21:53:49.427230 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.427176    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85f7d35-4bff-45d0-a392-2ecf056c2051-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xnbks\" (UID: \"a85f7d35-4bff-45d0-a392-2ecf056c2051\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 21:53:49.427230 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.427196    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a85f7d35-4bff-45d0-a392-2ecf056c2051-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xnbks\" (UID: \"a85f7d35-4bff-45d0-a392-2ecf056c2051\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 21:53:49.427412 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.427388    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a85f7d35-4bff-45d0-a392-2ecf056c2051-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xnbks\" (UID: \"a85f7d35-4bff-45d0-a392-2ecf056c2051\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 21:53:49.527906 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.527868    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a85f7d35-4bff-45d0-a392-2ecf056c2051-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xnbks\" (UID: \"a85f7d35-4bff-45d0-a392-2ecf056c2051\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 21:53:49.528106 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.527920    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrn2b\" (UniqueName: \"kubernetes.io/projected/a85f7d35-4bff-45d0-a392-2ecf056c2051-kube-api-access-wrn2b\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xnbks\" (UID: \"a85f7d35-4bff-45d0-a392-2ecf056c2051\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 21:53:49.528106 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.527960    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a85f7d35-4bff-45d0-a392-2ecf056c2051-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xnbks\" (UID: \"a85f7d35-4bff-45d0-a392-2ecf056c2051\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 21:53:49.528106 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.528001    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85f7d35-4bff-45d0-a392-2ecf056c2051-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xnbks\" (UID: \"a85f7d35-4bff-45d0-a392-2ecf056c2051\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 21:53:49.528106 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.528028    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a85f7d35-4bff-45d0-a392-2ecf056c2051-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xnbks\" (UID: \"a85f7d35-4bff-45d0-a392-2ecf056c2051\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 21:53:49.528106 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.528092    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a85f7d35-4bff-45d0-a392-2ecf056c2051-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xnbks\" (UID: \"a85f7d35-4bff-45d0-a392-2ecf056c2051\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 21:53:49.528389 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.528349    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a85f7d35-4bff-45d0-a392-2ecf056c2051-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xnbks\" (UID: \"a85f7d35-4bff-45d0-a392-2ecf056c2051\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 21:53:49.528445 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.528414    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a85f7d35-4bff-45d0-a392-2ecf056c2051-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xnbks\" (UID: \"a85f7d35-4bff-45d0-a392-2ecf056c2051\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 21:53:49.528667 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.528646    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a85f7d35-4bff-45d0-a392-2ecf056c2051-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xnbks\" (UID: \"a85f7d35-4bff-45d0-a392-2ecf056c2051\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 21:53:49.530587 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.530558    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a85f7d35-4bff-45d0-a392-2ecf056c2051-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xnbks\" (UID: \"a85f7d35-4bff-45d0-a392-2ecf056c2051\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 21:53:49.530799 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.530778    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a85f7d35-4bff-45d0-a392-2ecf056c2051-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xnbks\" (UID: \"a85f7d35-4bff-45d0-a392-2ecf056c2051\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 21:53:49.551673 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.551614    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrn2b\" (UniqueName: \"kubernetes.io/projected/a85f7d35-4bff-45d0-a392-2ecf056c2051-kube-api-access-wrn2b\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-xnbks\" (UID: \"a85f7d35-4bff-45d0-a392-2ecf056c2051\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 21:53:49.610722 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.610691    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 21:53:49.921161 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:49.921134    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"]
Oct 29 21:53:49.924453 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:53:49.924427    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85f7d35_4bff_45d0_a392_2ecf056c2051.slice/crio-f8604b88b2462c8e3186af1bb424b030042662b377c71d23d51f2d2866ea5906 WatchSource:0}: Error finding container f8604b88b2462c8e3186af1bb424b030042662b377c71d23d51f2d2866ea5906: Status 404 returned error can't find the container with id f8604b88b2462c8e3186af1bb424b030042662b377c71d23d51f2d2866ea5906
Oct 29 21:53:50.038559 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:50.038524    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks" event={"ID":"a85f7d35-4bff-45d0-a392-2ecf056c2051","Type":"ContainerStarted","Data":"f8604b88b2462c8e3186af1bb424b030042662b377c71d23d51f2d2866ea5906"}
Oct 29 21:53:50.040081 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:50.040057    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg" event={"ID":"70e1bafe-4b93-44d7-a179-44d5a82fd069","Type":"ContainerStarted","Data":"a92f6be278cc0b29a12799004e5c9c51988be39707593d239826af4b85477f02"}
Oct 29 21:53:50.084878 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:50.084822    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg" podStartSLOduration=0.955169089 podStartE2EDuration="2.084806903s" totalImagesPullingTime="1.129637814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:50.084251556 +0000 UTC m=+451.157208521" watchObservedRunningTime="2026-10-29 21:53:50.084806903 +0000 UTC m=+451.157763867"
Oct 29 21:53:50.373360 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:50.373329    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 21:53:51.046765 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:51.046715    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks" event={"ID":"a85f7d35-4bff-45d0-a392-2ecf056c2051","Type":"ContainerStarted","Data":"777b24d07bdb161dadea571e3afaa6fdf6c1a65131cede420dc4b452d001d171"}
Oct 29 21:53:51.091432 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:51.091386    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks" podStartSLOduration=1.875300163 podStartE2EDuration="2.091372159s" totalImagesPullingTime="216.071996ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:51.090557029 +0000 UTC m=+452.163513994" watchObservedRunningTime="2026-10-29 21:53:51.091372159 +0000 UTC m=+452.164329123"
Oct 29 21:53:51.613054 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:51.613009    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 21:53:54.526486 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:54.526454    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"]
Oct 29 21:53:54.530240 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:54.530215    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 21:53:54.556347 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:54.556324    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 29 21:53:54.557987 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:54.557963    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"]
Oct 29 21:53:54.684974 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:54.684939    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/696bd863-08fa-4694-843f-ab65f3ccfeed-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d\" (UID: \"696bd863-08fa-4694-843f-ab65f3ccfeed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 21:53:54.685100 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:54.684978    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/696bd863-08fa-4694-843f-ab65f3ccfeed-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d\" (UID: \"696bd863-08fa-4694-843f-ab65f3ccfeed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 21:53:54.685100 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:54.685015    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69ddz\" (UniqueName: \"kubernetes.io/projected/696bd863-08fa-4694-843f-ab65f3ccfeed-kube-api-access-69ddz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d\" (UID: \"696bd863-08fa-4694-843f-ab65f3ccfeed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 21:53:54.685181 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:54.685110    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/696bd863-08fa-4694-843f-ab65f3ccfeed-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d\" (UID: \"696bd863-08fa-4694-843f-ab65f3ccfeed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 21:53:54.685181 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:54.685154    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/696bd863-08fa-4694-843f-ab65f3ccfeed-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d\" (UID: \"696bd863-08fa-4694-843f-ab65f3ccfeed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 21:53:54.685181 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:54.685175    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/696bd863-08fa-4694-843f-ab65f3ccfeed-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d\" (UID: \"696bd863-08fa-4694-843f-ab65f3ccfeed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 21:53:54.786219 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:54.786143    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/696bd863-08fa-4694-843f-ab65f3ccfeed-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d\" (UID: \"696bd863-08fa-4694-843f-ab65f3ccfeed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 21:53:54.786219 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:54.786193    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/696bd863-08fa-4694-843f-ab65f3ccfeed-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d\" (UID: \"696bd863-08fa-4694-843f-ab65f3ccfeed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 21:53:54.786219 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:54.786217    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/696bd863-08fa-4694-843f-ab65f3ccfeed-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d\" (UID: \"696bd863-08fa-4694-843f-ab65f3ccfeed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 21:53:54.786500 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:54.786267    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/696bd863-08fa-4694-843f-ab65f3ccfeed-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d\" (UID: \"696bd863-08fa-4694-843f-ab65f3ccfeed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 21:53:54.786500 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:54.786304    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/696bd863-08fa-4694-843f-ab65f3ccfeed-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d\" (UID: \"696bd863-08fa-4694-843f-ab65f3ccfeed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 21:53:54.786500 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:54.786360    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69ddz\" (UniqueName: \"kubernetes.io/projected/696bd863-08fa-4694-843f-ab65f3ccfeed-kube-api-access-69ddz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d\" (UID: \"696bd863-08fa-4694-843f-ab65f3ccfeed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 21:53:54.786688 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:54.786663    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/696bd863-08fa-4694-843f-ab65f3ccfeed-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d\" (UID: \"696bd863-08fa-4694-843f-ab65f3ccfeed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 21:53:54.786750 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:54.786701    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/696bd863-08fa-4694-843f-ab65f3ccfeed-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d\" (UID: \"696bd863-08fa-4694-843f-ab65f3ccfeed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 21:53:54.786750 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:54.786722    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/696bd863-08fa-4694-843f-ab65f3ccfeed-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d\" (UID: \"696bd863-08fa-4694-843f-ab65f3ccfeed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 21:53:54.788339 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:54.788319    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/696bd863-08fa-4694-843f-ab65f3ccfeed-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d\" (UID: \"696bd863-08fa-4694-843f-ab65f3ccfeed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 21:53:54.788682 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:54.788662    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/696bd863-08fa-4694-843f-ab65f3ccfeed-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d\" (UID: \"696bd863-08fa-4694-843f-ab65f3ccfeed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 21:53:54.819342 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:54.819318    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69ddz\" (UniqueName: \"kubernetes.io/projected/696bd863-08fa-4694-843f-ab65f3ccfeed-kube-api-access-69ddz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d\" (UID: \"696bd863-08fa-4694-843f-ab65f3ccfeed\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 21:53:54.845316 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:54.845278    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 21:53:55.145054 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:55.145023    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"]
Oct 29 21:53:55.147204 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:53:55.147174    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod696bd863_08fa_4694_843f_ab65f3ccfeed.slice/crio-33f739d9094d25db6ae86ea151bea7f42be9fe99856b54427a73d30ccc10681b WatchSource:0}: Error finding container 33f739d9094d25db6ae86ea151bea7f42be9fe99856b54427a73d30ccc10681b: Status 404 returned error can't find the container with id 33f739d9094d25db6ae86ea151bea7f42be9fe99856b54427a73d30ccc10681b
Oct 29 21:53:56.084491 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:56.084450    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d" event={"ID":"696bd863-08fa-4694-843f-ab65f3ccfeed","Type":"ContainerStarted","Data":"d1b36b21d8fcfd7f39175febe98b8728bb321d833e1d34c8d04a72ebeded8792"}
Oct 29 21:53:56.085087 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:56.085061    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d" event={"ID":"696bd863-08fa-4694-843f-ab65f3ccfeed","Type":"ContainerStarted","Data":"33f739d9094d25db6ae86ea151bea7f42be9fe99856b54427a73d30ccc10681b"}
Oct 29 21:53:56.147313 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:56.146513    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d" podStartSLOduration=1.920039016 podStartE2EDuration="2.146492112s" totalImagesPullingTime="226.453096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:53:56.145962991 +0000 UTC m=+457.218919955" watchObservedRunningTime="2026-10-29 21:53:56.146492112 +0000 UTC m=+457.219449078"
Oct 29 21:53:56.845584 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:56.845545    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 21:53:58.373580 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:58.373538    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 21:53:58.386882 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:58.386851    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 21:53:59.146673 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:59.146630    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 21:53:59.616312 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:59.615360    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 21:53:59.648363 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:53:59.648336    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 21:54:00.154425 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:00.154393    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 21:54:04.846556 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:04.845618    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 21:54:04.859914 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:04.859886    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 21:54:05.173427 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:05.173401    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 21:54:09.719259 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:54:09.719222    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:54:12.812344 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:12.812312    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"]
Oct 29 21:54:12.815984 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:12.815960    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:12.847581 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:12.847555    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"]
Oct 29 21:54:12.922119 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:12.922095    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:12.922240 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:12.922127    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:12.922240 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:12.922155    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:12.922338 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:12.922271    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:12.922338 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:12.922302    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmkzc\" (UniqueName: \"kubernetes.io/projected/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-kube-api-access-bmkzc\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:12.922338 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:12.922323    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:12.922445 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:12.922423    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:12.922479 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:12.922453    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:12.922479 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:12.922473    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:13.023172 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:13.023146    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:13.023270 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:13.023191    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:13.023270 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:13.023221    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:13.023270 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:13.023248    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:13.023423 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:13.023325    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:13.023423 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:13.023354    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:13.023423 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:13.023391    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:13.023627 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:13.023605    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:13.023817 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:13.023630    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:13.023944 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:13.023922    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:13.024015 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:13.023808    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:13.024015 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:13.023761    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:13.024015 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:13.023997    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:13.024148 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:13.024049    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmkzc\" (UniqueName: \"kubernetes.io/projected/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-kube-api-access-bmkzc\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:13.025655 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:13.025628    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:13.025735 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:13.025709    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:13.046474 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:13.046449    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:13.046721 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:13.046701    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmkzc\" (UniqueName: \"kubernetes.io/projected/aacf5dff-ff07-4c5c-a98f-db3d55beb6b5-kube-api-access-bmkzc\") pod \"maas-default-gateway-openshift-default-769df7846c-krsk7\" (UID: \"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:13.129506 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:13.129460    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:13.443440 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:13.443413    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"]
Oct 29 21:54:13.446514 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:54:13.446480    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaacf5dff_ff07_4c5c_a98f_db3d55beb6b5.slice/crio-e56e8aaa26b96e1dd73a91f16de2ec973fa60d03e8ebd800aca7afe25e88ba19 WatchSource:0}: Error finding container e56e8aaa26b96e1dd73a91f16de2ec973fa60d03e8ebd800aca7afe25e88ba19: Status 404 returned error can't find the container with id e56e8aaa26b96e1dd73a91f16de2ec973fa60d03e8ebd800aca7afe25e88ba19
Oct 29 21:54:13.448825 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:13.448789    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:54:13.448885 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:13.448874    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:54:13.448930 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:13.448918    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:54:14.212994 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:14.212950    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7" event={"ID":"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5","Type":"ContainerStarted","Data":"79d04c19abb7781016ad13b5d2def6a819a12043b5cabf3248dae1197b8d78c1"}
Oct 29 21:54:14.213551 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:14.213054    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7" event={"ID":"aacf5dff-ff07-4c5c-a98f-db3d55beb6b5","Type":"ContainerStarted","Data":"e56e8aaa26b96e1dd73a91f16de2ec973fa60d03e8ebd800aca7afe25e88ba19"}
Oct 29 21:54:14.297615 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:14.297550    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7" podStartSLOduration=2.297534639 podStartE2EDuration="2.297534639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:54:14.2704003 +0000 UTC m=+475.343357263" watchObservedRunningTime="2026-10-29 21:54:14.297534639 +0000 UTC m=+475.370491602"
Oct 29 21:54:15.130428 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:15.130396    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:15.131558 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:15.131535    4962 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-krsk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 29 21:54:15.131681 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:15.131587    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7" podUID="aacf5dff-ff07-4c5c-a98f-db3d55beb6b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 29 21:54:16.130215 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:16.130183    4962 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-krsk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 29 21:54:16.130627 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:16.130261    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7" podUID="aacf5dff-ff07-4c5c-a98f-db3d55beb6b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 29 21:54:17.130644 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:17.130610    4962 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-krsk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 29 21:54:17.131024 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:17.130669    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7" podUID="aacf5dff-ff07-4c5c-a98f-db3d55beb6b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 29 21:54:18.130110 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:18.130078    4962 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-krsk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 29 21:54:18.130359 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:18.130136    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7" podUID="aacf5dff-ff07-4c5c-a98f-db3d55beb6b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 29 21:54:19.130766 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:19.130731    4962 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-krsk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 29 21:54:19.131064 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:19.130791    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7" podUID="aacf5dff-ff07-4c5c-a98f-db3d55beb6b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 29 21:54:20.130052 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:20.130020    4962 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-krsk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body=
Oct 29 21:54:20.130250 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:20.130074    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7" podUID="aacf5dff-ff07-4c5c-a98f-db3d55beb6b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused"
Oct 29 21:54:21.910640 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:21.910600    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:21.911767 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:21.911739    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:21.912572 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:21.912548    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-krsk7"
Oct 29 21:54:39.718867 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:54:39.718789    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:54:58.989695 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:58.989656    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"]
Oct 29 21:54:58.993846 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:58.993822    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.010763 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.010741    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f5bae148-openshift-default-dockercfg-kk9n4\""
Oct 29 21:54:59.019376 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.019351    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-credential-socket\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.019486 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.019406    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-workload-socket\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.019486 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.019476    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-token\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.019604 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.019503    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-podinfo\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.019604 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.019565    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-envoy\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.019604 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.019597    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istiod-ca-cert\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.019930 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.019908    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-data\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.020030 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.019939    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-workload-certs\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.020030 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.020011    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx6jz\" (UniqueName: \"kubernetes.io/projected/590636b8-d3a4-4c12-9c93-7755dbdf4f37-kube-api-access-zx6jz\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.024515 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.024494    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"]
Oct 29 21:54:59.120629 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.120591    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-data\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.120810 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.120641    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-workload-certs\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.120810 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.120693    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx6jz\" (UniqueName: \"kubernetes.io/projected/590636b8-d3a4-4c12-9c93-7755dbdf4f37-kube-api-access-zx6jz\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.120810 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.120737    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-credential-socket\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.120810 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.120776    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-workload-socket\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.120810 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.120808    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-token\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.121098 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.120834    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-podinfo\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.121098 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.120889    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-envoy\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.121098 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.120929    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istiod-ca-cert\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.121098 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.120994    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-data\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.121410 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.121385    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-workload-socket\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.121679 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.121652    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istiod-ca-cert\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.121874 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.121857    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-credential-socket\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.121952 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.121943    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-workload-certs\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.124040 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.124016    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-envoy\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.124153 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.124116    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-podinfo\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.146533 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.146504    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-token\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.147425 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.147406    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx6jz\" (UniqueName: \"kubernetes.io/projected/590636b8-d3a4-4c12-9c93-7755dbdf4f37-kube-api-access-zx6jz\") pod \"e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") " pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.263549 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.263474    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"]
Oct 29 21:54:59.268448 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.268428    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.285489 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.285465    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-64505fc5-openshift-default-dockercfg-ftvt4\""
Oct 29 21:54:59.292658 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.292633    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"]
Oct 29 21:54:59.307795 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.307775    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:54:59.322638 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.322407    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-workload-socket\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.322638 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.322480    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-data\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.322638 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.322524    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-credential-socket\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.322638 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.322563    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-token\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.322638 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.322596    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-workload-certs\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.322915 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.322783    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfx9w\" (UniqueName: \"kubernetes.io/projected/ac4fac5d-8644-4da7-82a0-aea07ee3f699-kube-api-access-xfx9w\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.322915 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.322827    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-podinfo\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.322915 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.322870    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-envoy\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.323022 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.322992    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istiod-ca-cert\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.423928 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.423893    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-podinfo\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.424093 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.423940    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-envoy\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.424093 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.423987    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istiod-ca-cert\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.424093 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.424012    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-workload-socket\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.424093 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.424066    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-data\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.424320 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.424105    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-credential-socket\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.424320 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.424143    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-token\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.424320 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.424202    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-workload-certs\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.424320 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.424226    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfx9w\" (UniqueName: \"kubernetes.io/projected/ac4fac5d-8644-4da7-82a0-aea07ee3f699-kube-api-access-xfx9w\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.424530 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.424508    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-workload-socket\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.424616 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.424594    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-credential-socket\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.424794 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.424773    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-workload-certs\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.424866 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.424851    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istiod-ca-cert\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.424920 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.424890    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-data\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.426385 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.426362    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-envoy\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.426748 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.426721    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-podinfo\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.447412 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.447385    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-token\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.447939 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.447883    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfx9w\" (UniqueName: \"kubernetes.io/projected/ac4fac5d-8644-4da7-82a0-aea07ee3f699-kube-api-access-xfx9w\") pod \"e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") " pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.591368 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.589479    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:54:59.661904 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.661870    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"]
Oct 29 21:54:59.673484 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.672950    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:54:59.673484 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.673030    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:54:59.673484 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.673078    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:54:59.790329 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.788268    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"]
Oct 29 21:54:59.792544 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.792514    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.808421 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.808206    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-8fba532a-gw-openshift-default-dockercfg-bfxwc\""
Oct 29 21:54:59.817881 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.817827    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"]
Oct 29 21:54:59.829922 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.829894    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-podinfo\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.830047 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.829941    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46cqj\" (UniqueName: \"kubernetes.io/projected/10a9a83c-2b74-4cba-8011-63a7f4f2f134-kube-api-access-46cqj\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.830047 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.829971    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-credential-socket\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.830154 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.830079    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-envoy\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.830216 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.830194    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-token\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.830269 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.830237    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-workload-socket\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.830331 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.830271    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istiod-ca-cert\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.830392 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.830381    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-data\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.830450 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.830428    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-workload-certs\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.933550 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.931032    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46cqj\" (UniqueName: \"kubernetes.io/projected/10a9a83c-2b74-4cba-8011-63a7f4f2f134-kube-api-access-46cqj\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.933550 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.931093    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-credential-socket\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.933550 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.931121    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-envoy\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.933550 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.931190    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-token\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.933550 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.931226    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-workload-socket\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.933550 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.931269    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istiod-ca-cert\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.937829 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.937792    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-data\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.937978 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.937843    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-workload-certs\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.937978 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.937870    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-podinfo\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.938734 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.938706    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"]
Oct 29 21:54:59.942789 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.942677    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-credential-socket\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.943221 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.943196    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istiod-ca-cert\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.943381 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.943359    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-workload-certs\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.943444 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.943383    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-workload-socket\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.944408 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.943840    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-data\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.944408 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.944068    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-podinfo\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.949417 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.946867    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-envoy\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.950343 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:54:59.950256    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4fac5d_8644_4da7_82a0_aea07ee3f699.slice/crio-42957d35c93166c022dab2e6f7d120026a9f078370777e345982183bbc2f5aba WatchSource:0}: Error finding container 42957d35c93166c022dab2e6f7d120026a9f078370777e345982183bbc2f5aba: Status 404 returned error can't find the container with id 42957d35c93166c022dab2e6f7d120026a9f078370777e345982183bbc2f5aba
Oct 29 21:54:59.953184 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.953152    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:54:59.953258 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.953234    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:54:59.953368 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.953335    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:54:59.965068 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.965049    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46cqj\" (UniqueName: \"kubernetes.io/projected/10a9a83c-2b74-4cba-8011-63a7f4f2f134-kube-api-access-46cqj\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:54:59.966081 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:54:59.966061    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-token\") pod \"e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") " pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:55:00.108323 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:00.108257    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:55:00.418230 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:00.418188    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm" event={"ID":"590636b8-d3a4-4c12-9c93-7755dbdf4f37","Type":"ContainerStarted","Data":"9cecdea73b7afddf02f82b32c4b7a3dceae967f59558bd45f1c9ba2b1ff8dc79"}
Oct 29 21:55:00.418409 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:00.418249    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm" event={"ID":"590636b8-d3a4-4c12-9c93-7755dbdf4f37","Type":"ContainerStarted","Data":"6b7ba12b50139be7566e5cd194da167156673d3d5f083bbffb10b3eadce19d87"}
Oct 29 21:55:00.419866 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:00.419840    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc" event={"ID":"ac4fac5d-8644-4da7-82a0-aea07ee3f699","Type":"ContainerStarted","Data":"1c1e0a166dd6749274c22ddd84d5179566f0977cbe88af58d7de628ce5d6b763"}
Oct 29 21:55:00.419985 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:00.419886    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc" event={"ID":"ac4fac5d-8644-4da7-82a0-aea07ee3f699","Type":"ContainerStarted","Data":"42957d35c93166c022dab2e6f7d120026a9f078370777e345982183bbc2f5aba"}
Oct 29 21:55:00.445737 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:00.445716    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"]
Oct 29 21:55:00.447871 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:55:00.447846    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a9a83c_2b74_4cba_8011_63a7f4f2f134.slice/crio-512ab4a5629739dc0d10c781e4ebe4b0a41f5d9e8b7d391f52cae51602ea1a69 WatchSource:0}: Error finding container 512ab4a5629739dc0d10c781e4ebe4b0a41f5d9e8b7d391f52cae51602ea1a69: Status 404 returned error can't find the container with id 512ab4a5629739dc0d10c781e4ebe4b0a41f5d9e8b7d391f52cae51602ea1a69
Oct 29 21:55:00.450475 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:00.450445    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:55:00.450580 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:00.450516    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:55:00.450580 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:00.450562    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:55:00.466340 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:00.466282    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm" podStartSLOduration=2.4662713529999998 podStartE2EDuration="2.466271353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:00.46444185 +0000 UTC m=+521.537398813" watchObservedRunningTime="2026-10-29 21:55:00.466271353 +0000 UTC m=+521.539228377"
Oct 29 21:55:00.510177 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:00.510137    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc" podStartSLOduration=1.510124115 podStartE2EDuration="1.510124115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:00.508996095 +0000 UTC m=+521.581953060" watchObservedRunningTime="2026-10-29 21:55:00.510124115 +0000 UTC m=+521.583081138"
Oct 29 21:55:00.590342 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:00.590311    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:55:01.313808 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:01.312661    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:55:01.318633 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:01.318604    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:55:01.425888 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:01.425853    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh" event={"ID":"10a9a83c-2b74-4cba-8011-63a7f4f2f134","Type":"ContainerStarted","Data":"42d21d24e8726835a75e170c7a68ea46b03430e7e24ddfe6a2fce7c459e71fa8"}
Oct 29 21:55:01.426037 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:01.425929    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh" event={"ID":"10a9a83c-2b74-4cba-8011-63a7f4f2f134","Type":"ContainerStarted","Data":"512ab4a5629739dc0d10c781e4ebe4b0a41f5d9e8b7d391f52cae51602ea1a69"}
Oct 29 21:55:01.426265 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:01.426241    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:55:01.427013 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:01.426993    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:55:01.473452 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:01.473409    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh" podStartSLOduration=2.473396606 podStartE2EDuration="2.473396606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:01.472316352 +0000 UTC m=+522.545273312" watchObservedRunningTime="2026-10-29 21:55:01.473396606 +0000 UTC m=+522.546353569"
Oct 29 21:55:01.594675 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:01.594603    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:55:02.109079 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:02.109042    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:55:02.114140 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:02.114114    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:55:02.431245 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:02.431209    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:55:02.431687 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:02.431270    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:55:02.431894 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:02.431874    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:55:02.432440 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:02.432421    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:55:03.782604 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:03.782568    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr"]
Oct 29 21:55:03.785909 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:03.785894    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr"
Oct 29 21:55:03.813259 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:03.813002    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f5bae148-serving-cert\""
Oct 29 21:55:03.813508 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:03.813070    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 21:55:03.813600 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:03.813163    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2qkjk\""
Oct 29 21:55:03.813600 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:03.813343    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 29 21:55:03.813600 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:03.813395    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 21:55:03.815165 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:03.815139    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr"]
Oct 29 21:55:03.921603 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:03.921572    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d91e7562-34b9-4350-896a-c95260f5c07a-openshift-service-ca\") pod \"maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr\" (UID: \"d91e7562-34b9-4350-896a-c95260f5c07a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr"
Oct 29 21:55:03.921754 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:03.921706    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d91e7562-34b9-4350-896a-c95260f5c07a-maas-api-tls\") pod \"maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr\" (UID: \"d91e7562-34b9-4350-896a-c95260f5c07a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr"
Oct 29 21:55:03.921754 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:03.921732    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5z26\" (UniqueName: \"kubernetes.io/projected/d91e7562-34b9-4350-896a-c95260f5c07a-kube-api-access-z5z26\") pod \"maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr\" (UID: \"d91e7562-34b9-4350-896a-c95260f5c07a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr"
Oct 29 21:55:03.921844 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:03.921833    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91e7562-34b9-4350-896a-c95260f5c07a-metrics-certs\") pod \"maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr\" (UID: \"d91e7562-34b9-4350-896a-c95260f5c07a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr"
Oct 29 21:55:04.022917 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.022884    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91e7562-34b9-4350-896a-c95260f5c07a-metrics-certs\") pod \"maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr\" (UID: \"d91e7562-34b9-4350-896a-c95260f5c07a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr"
Oct 29 21:55:04.023077 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.022930    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d91e7562-34b9-4350-896a-c95260f5c07a-openshift-service-ca\") pod \"maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr\" (UID: \"d91e7562-34b9-4350-896a-c95260f5c07a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr"
Oct 29 21:55:04.023077 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.023010    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d91e7562-34b9-4350-896a-c95260f5c07a-maas-api-tls\") pod \"maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr\" (UID: \"d91e7562-34b9-4350-896a-c95260f5c07a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr"
Oct 29 21:55:04.023077 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.023031    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5z26\" (UniqueName: \"kubernetes.io/projected/d91e7562-34b9-4350-896a-c95260f5c07a-kube-api-access-z5z26\") pod \"maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr\" (UID: \"d91e7562-34b9-4350-896a-c95260f5c07a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr"
Oct 29 21:55:04.023569 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.023532    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6"]
Oct 29 21:55:04.023721 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.023701    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d91e7562-34b9-4350-896a-c95260f5c07a-openshift-service-ca\") pod \"maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr\" (UID: \"d91e7562-34b9-4350-896a-c95260f5c07a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr"
Oct 29 21:55:04.025302 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.025273    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91e7562-34b9-4350-896a-c95260f5c07a-metrics-certs\") pod \"maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr\" (UID: \"d91e7562-34b9-4350-896a-c95260f5c07a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr"
Oct 29 21:55:04.025418 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.025387    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d91e7562-34b9-4350-896a-c95260f5c07a-maas-api-tls\") pod \"maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr\" (UID: \"d91e7562-34b9-4350-896a-c95260f5c07a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr"
Oct 29 21:55:04.027271 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.027255    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6"
Oct 29 21:55:04.064834 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.064812    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f5bae148\""
Oct 29 21:55:04.066801 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.066778    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6"]
Oct 29 21:55:04.077116 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.077088    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5z26\" (UniqueName: \"kubernetes.io/projected/d91e7562-34b9-4350-896a-c95260f5c07a-kube-api-access-z5z26\") pod \"maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr\" (UID: \"d91e7562-34b9-4350-896a-c95260f5c07a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr"
Oct 29 21:55:04.084187 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.084159    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l"]
Oct 29 21:55:04.087924 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.087906    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l"
Oct 29 21:55:04.098436 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.098415    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr"
Oct 29 21:55:04.112931 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.112906    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f5bae148-dockercfg-dvd8v\""
Oct 29 21:55:04.119759 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.119735    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l"]
Oct 29 21:55:04.128953 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.124376    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp5xn\" (UniqueName: \"kubernetes.io/projected/ad9d9073-c42c-4c70-90a5-5971211c41b2-kube-api-access-xp5xn\") pod \"payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6\" (UID: \"ad9d9073-c42c-4c70-90a5-5971211c41b2\") " pod="openshift-ingress/payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6"
Oct 29 21:55:04.128953 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.124530    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad9d9073-c42c-4c70-90a5-5971211c41b2-plugins-config-volume\") pod \"payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6\" (UID: \"ad9d9073-c42c-4c70-90a5-5971211c41b2\") " pod="openshift-ingress/payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6"
Oct 29 21:55:04.227322 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.226834    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb1dcd64-e3c0-4ac0-8941-c0845dba1979-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l\" (UID: \"eb1dcd64-e3c0-4ac0-8941-c0845dba1979\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l"
Oct 29 21:55:04.227322 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.226896    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlhw8\" (UniqueName: \"kubernetes.io/projected/eb1dcd64-e3c0-4ac0-8941-c0845dba1979-kube-api-access-qlhw8\") pod \"payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l\" (UID: \"eb1dcd64-e3c0-4ac0-8941-c0845dba1979\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l"
Oct 29 21:55:04.227322 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.226978    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp5xn\" (UniqueName: \"kubernetes.io/projected/ad9d9073-c42c-4c70-90a5-5971211c41b2-kube-api-access-xp5xn\") pod \"payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6\" (UID: \"ad9d9073-c42c-4c70-90a5-5971211c41b2\") " pod="openshift-ingress/payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6"
Oct 29 21:55:04.227322 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.227027    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad9d9073-c42c-4c70-90a5-5971211c41b2-plugins-config-volume\") pod \"payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6\" (UID: \"ad9d9073-c42c-4c70-90a5-5971211c41b2\") " pod="openshift-ingress/payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6"
Oct 29 21:55:04.240517 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.230487    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad9d9073-c42c-4c70-90a5-5971211c41b2-plugins-config-volume\") pod \"payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6\" (UID: \"ad9d9073-c42c-4c70-90a5-5971211c41b2\") " pod="openshift-ingress/payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6"
Oct 29 21:55:04.272198 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.265363    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp5xn\" (UniqueName: \"kubernetes.io/projected/ad9d9073-c42c-4c70-90a5-5971211c41b2-kube-api-access-xp5xn\") pod \"payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6\" (UID: \"ad9d9073-c42c-4c70-90a5-5971211c41b2\") " pod="openshift-ingress/payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6"
Oct 29 21:55:04.327788 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.327710    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlhw8\" (UniqueName: \"kubernetes.io/projected/eb1dcd64-e3c0-4ac0-8941-c0845dba1979-kube-api-access-qlhw8\") pod \"payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l\" (UID: \"eb1dcd64-e3c0-4ac0-8941-c0845dba1979\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l"
Oct 29 21:55:04.327926 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.327854    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb1dcd64-e3c0-4ac0-8941-c0845dba1979-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l\" (UID: \"eb1dcd64-e3c0-4ac0-8941-c0845dba1979\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l"
Oct 29 21:55:04.328846 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.328826    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb1dcd64-e3c0-4ac0-8941-c0845dba1979-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l\" (UID: \"eb1dcd64-e3c0-4ac0-8941-c0845dba1979\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l"
Oct 29 21:55:04.350992 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.350968    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlhw8\" (UniqueName: \"kubernetes.io/projected/eb1dcd64-e3c0-4ac0-8941-c0845dba1979-kube-api-access-qlhw8\") pod \"payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l\" (UID: \"eb1dcd64-e3c0-4ac0-8941-c0845dba1979\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l"
Oct 29 21:55:04.351742 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.351707    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6"
Oct 29 21:55:04.404183 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.402412    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l"
Oct 29 21:55:04.405490 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.405469    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr"]
Oct 29 21:55:04.408599 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:55:04.408559    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91e7562_34b9_4350_896a_c95260f5c07a.slice/crio-403c616c85beb688aae992f306e51d3e4ce6438ba81675d2f562ad54525a596d WatchSource:0}: Error finding container 403c616c85beb688aae992f306e51d3e4ce6438ba81675d2f562ad54525a596d: Status 404 returned error can't find the container with id 403c616c85beb688aae992f306e51d3e4ce6438ba81675d2f562ad54525a596d
Oct 29 21:55:04.443333 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.441009    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr" event={"ID":"d91e7562-34b9-4350-896a-c95260f5c07a","Type":"ContainerStarted","Data":"403c616c85beb688aae992f306e51d3e4ce6438ba81675d2f562ad54525a596d"}
Oct 29 21:55:04.657992 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.657929    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6"]
Oct 29 21:55:04.714264 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:04.714235    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l"]
Oct 29 21:55:04.717643 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:55:04.717460    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1dcd64_e3c0_4ac0_8941_c0845dba1979.slice/crio-7d5ff22c954bee2dcbf1c3dc248068919428da253f6ca257bdb0ef6f37d1854b WatchSource:0}: Error finding container 7d5ff22c954bee2dcbf1c3dc248068919428da253f6ca257bdb0ef6f37d1854b: Status 404 returned error can't find the container with id 7d5ff22c954bee2dcbf1c3dc248068919428da253f6ca257bdb0ef6f37d1854b
Oct 29 21:55:05.446626 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.446590    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr" event={"ID":"d91e7562-34b9-4350-896a-c95260f5c07a","Type":"ContainerStarted","Data":"ae2f1fbec7120f85dfe25017f780112ab7486fab839439328db484933c58bceb"}
Oct 29 21:55:05.447059 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.446657    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr"
Oct 29 21:55:05.448351 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.448316    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6" event={"ID":"ad9d9073-c42c-4c70-90a5-5971211c41b2","Type":"ContainerStarted","Data":"6dd017dcc5533d9924ef61dbec91c12c1d9696dd6db7e853ee79cd0a1534a599"}
Oct 29 21:55:05.448465 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.448368    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6" event={"ID":"ad9d9073-c42c-4c70-90a5-5971211c41b2","Type":"ContainerStarted","Data":"9e7bed16128698ae1f5bb0345feaf5ef7f54e020c75ab7f27814f37197ab97dc"}
Oct 29 21:55:05.448465 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.448399    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6"
Oct 29 21:55:05.449939 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.449912    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l" event={"ID":"eb1dcd64-e3c0-4ac0-8941-c0845dba1979","Type":"ContainerStarted","Data":"fd898439e4264c4f0d52994ddab3a3e7562d846b96fa30993cf37f092e3a08d7"}
Oct 29 21:55:05.450054 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.449955    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l" event={"ID":"eb1dcd64-e3c0-4ac0-8941-c0845dba1979","Type":"ContainerStarted","Data":"7d5ff22c954bee2dcbf1c3dc248068919428da253f6ca257bdb0ef6f37d1854b"}
Oct 29 21:55:05.450114 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.450062    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l"
Oct 29 21:55:05.537379 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.537321    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr" podStartSLOduration=1.805956178 podStartE2EDuration="2.537308202s" totalImagesPullingTime="731.352024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:05.492813953 +0000 UTC m=+526.565770916" watchObservedRunningTime="2026-10-29 21:55:05.537308202 +0000 UTC m=+526.610265160"
Oct 29 21:55:05.538092 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.538054    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6" podStartSLOduration=2.538044975 podStartE2EDuration="2.538044975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:05.535964235 +0000 UTC m=+526.608921219" watchObservedRunningTime="2026-10-29 21:55:05.538044975 +0000 UTC m=+526.611001939"
Oct 29 21:55:05.580798 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.580748    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l" podStartSLOduration=1.58073583 podStartE2EDuration="1.58073583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:05.578944859 +0000 UTC m=+526.651901823" watchObservedRunningTime="2026-10-29 21:55:05.58073583 +0000 UTC m=+526.653692794"
Oct 29 21:55:05.848359 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.814353    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx"]
Oct 29 21:55:05.873633 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.858629    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx"
Oct 29 21:55:05.873633 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.873439    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-64505fc5\""
Oct 29 21:55:05.878718 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.878687    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4"]
Oct 29 21:55:05.888103 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.888080    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx"]
Oct 29 21:55:05.888329 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.888301    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4"]
Oct 29 21:55:05.888552 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.888538    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4"
Oct 29 21:55:05.902775 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.902711    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-64505fc5-dockercfg-fsrr5\""
Oct 29 21:55:05.904585 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.903723    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-64505fc5-serving-cert\""
Oct 29 21:55:05.957077 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.957050    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txxbr\" (UniqueName: \"kubernetes.io/projected/0a47e418-7c3c-4f35-a423-ac4296c9c8a7-kube-api-access-txxbr\") pod \"payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx\" (UID: \"0a47e418-7c3c-4f35-a423-ac4296c9c8a7\") " pod="openshift-ingress/payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx"
Oct 29 21:55:05.957202 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.957105    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d62760e8-caa9-49a1-976d-c56196ecd567-metrics-certs\") pod \"maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4\" (UID: \"d62760e8-caa9-49a1-976d-c56196ecd567\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4"
Oct 29 21:55:05.957254 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.957213    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a47e418-7c3c-4f35-a423-ac4296c9c8a7-plugins-config-volume\") pod \"payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx\" (UID: \"0a47e418-7c3c-4f35-a423-ac4296c9c8a7\") " pod="openshift-ingress/payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx"
Oct 29 21:55:05.957327 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.957252    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbxcc\" (UniqueName: \"kubernetes.io/projected/d62760e8-caa9-49a1-976d-c56196ecd567-kube-api-access-mbxcc\") pod \"maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4\" (UID: \"d62760e8-caa9-49a1-976d-c56196ecd567\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4"
Oct 29 21:55:05.957327 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.957276    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d62760e8-caa9-49a1-976d-c56196ecd567-maas-api-tls\") pod \"maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4\" (UID: \"d62760e8-caa9-49a1-976d-c56196ecd567\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4"
Oct 29 21:55:05.957421 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:05.957406    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62760e8-caa9-49a1-976d-c56196ecd567-openshift-service-ca\") pod \"maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4\" (UID: \"d62760e8-caa9-49a1-976d-c56196ecd567\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4"
Oct 29 21:55:06.058650 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:06.058607    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a47e418-7c3c-4f35-a423-ac4296c9c8a7-plugins-config-volume\") pod \"payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx\" (UID: \"0a47e418-7c3c-4f35-a423-ac4296c9c8a7\") " pod="openshift-ingress/payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx"
Oct 29 21:55:06.058650 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:06.058667    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbxcc\" (UniqueName: \"kubernetes.io/projected/d62760e8-caa9-49a1-976d-c56196ecd567-kube-api-access-mbxcc\") pod \"maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4\" (UID: \"d62760e8-caa9-49a1-976d-c56196ecd567\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4"
Oct 29 21:55:06.058948 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:06.058694    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d62760e8-caa9-49a1-976d-c56196ecd567-maas-api-tls\") pod \"maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4\" (UID: \"d62760e8-caa9-49a1-976d-c56196ecd567\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4"
Oct 29 21:55:06.058948 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:06.058741    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62760e8-caa9-49a1-976d-c56196ecd567-openshift-service-ca\") pod \"maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4\" (UID: \"d62760e8-caa9-49a1-976d-c56196ecd567\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4"
Oct 29 21:55:06.058948 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:06.058781    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txxbr\" (UniqueName: \"kubernetes.io/projected/0a47e418-7c3c-4f35-a423-ac4296c9c8a7-kube-api-access-txxbr\") pod \"payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx\" (UID: \"0a47e418-7c3c-4f35-a423-ac4296c9c8a7\") " pod="openshift-ingress/payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx"
Oct 29 21:55:06.058948 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:06.058831    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d62760e8-caa9-49a1-976d-c56196ecd567-metrics-certs\") pod \"maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4\" (UID: \"d62760e8-caa9-49a1-976d-c56196ecd567\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4"
Oct 29 21:55:06.060330 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:06.060265    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62760e8-caa9-49a1-976d-c56196ecd567-openshift-service-ca\") pod \"maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4\" (UID: \"d62760e8-caa9-49a1-976d-c56196ecd567\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4"
Oct 29 21:55:06.060564 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:06.060544    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a47e418-7c3c-4f35-a423-ac4296c9c8a7-plugins-config-volume\") pod \"payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx\" (UID: \"0a47e418-7c3c-4f35-a423-ac4296c9c8a7\") " pod="openshift-ingress/payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx"
Oct 29 21:55:06.062055 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:06.062034    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d62760e8-caa9-49a1-976d-c56196ecd567-maas-api-tls\") pod \"maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4\" (UID: \"d62760e8-caa9-49a1-976d-c56196ecd567\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4"
Oct 29 21:55:06.062201 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:06.062147    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d62760e8-caa9-49a1-976d-c56196ecd567-metrics-certs\") pod \"maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4\" (UID: \"d62760e8-caa9-49a1-976d-c56196ecd567\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4"
Oct 29 21:55:06.082856 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:06.082833    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txxbr\" (UniqueName: \"kubernetes.io/projected/0a47e418-7c3c-4f35-a423-ac4296c9c8a7-kube-api-access-txxbr\") pod \"payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx\" (UID: \"0a47e418-7c3c-4f35-a423-ac4296c9c8a7\") " pod="openshift-ingress/payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx"
Oct 29 21:55:06.082953 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:06.082875    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbxcc\" (UniqueName: \"kubernetes.io/projected/d62760e8-caa9-49a1-976d-c56196ecd567-kube-api-access-mbxcc\") pod \"maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4\" (UID: \"d62760e8-caa9-49a1-976d-c56196ecd567\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4"
Oct 29 21:55:06.193516 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:06.193492    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx"
Oct 29 21:55:06.212506 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:06.212486    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4"
Oct 29 21:55:06.495232 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:06.495205    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx"]
Oct 29 21:55:06.496949 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:55:06.496923    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a47e418_7c3c_4f35_a423_ac4296c9c8a7.slice/crio-9e4d5e2fcf3e2fff44e46783440e65dd491066555e8c78df745b115ce1442976 WatchSource:0}: Error finding container 9e4d5e2fcf3e2fff44e46783440e65dd491066555e8c78df745b115ce1442976: Status 404 returned error can't find the container with id 9e4d5e2fcf3e2fff44e46783440e65dd491066555e8c78df745b115ce1442976
Oct 29 21:55:06.524909 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:06.524890    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4"]
Oct 29 21:55:06.530466 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:55:06.530443    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd62760e8_caa9_49a1_976d_c56196ecd567.slice/crio-88cdc61c874f1875bb915ffcc84282d53ede6dff3489f76c097d07c5d6cc8ed1 WatchSource:0}: Error finding container 88cdc61c874f1875bb915ffcc84282d53ede6dff3489f76c097d07c5d6cc8ed1: Status 404 returned error can't find the container with id 88cdc61c874f1875bb915ffcc84282d53ede6dff3489f76c097d07c5d6cc8ed1
Oct 29 21:55:07.461260 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:07.461222    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4" event={"ID":"d62760e8-caa9-49a1-976d-c56196ecd567","Type":"ContainerStarted","Data":"9ca4cb84c2aa6cb553fe1e30a966dc8b56c4f766e05c5cc18d18ff8713e93e13"}
Oct 29 21:55:07.461470 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:07.461283    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4" event={"ID":"d62760e8-caa9-49a1-976d-c56196ecd567","Type":"ContainerStarted","Data":"88cdc61c874f1875bb915ffcc84282d53ede6dff3489f76c097d07c5d6cc8ed1"}
Oct 29 21:55:07.461470 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:07.461334    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4"
Oct 29 21:55:07.462935 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:07.462905    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx" event={"ID":"0a47e418-7c3c-4f35-a423-ac4296c9c8a7","Type":"ContainerStarted","Data":"96e15802c93aa45b0470df239550af358d9c3a013ade0f52a51d6b35fecebf0a"}
Oct 29 21:55:07.463045 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:07.462954    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx"
Oct 29 21:55:07.463045 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:07.462976    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx" event={"ID":"0a47e418-7c3c-4f35-a423-ac4296c9c8a7","Type":"ContainerStarted","Data":"9e4d5e2fcf3e2fff44e46783440e65dd491066555e8c78df745b115ce1442976"}
Oct 29 21:55:07.509545 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:07.509502    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4" podStartSLOduration=1.990901496 podStartE2EDuration="2.509488403s" totalImagesPullingTime="518.586907ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:07.508234045 +0000 UTC m=+528.581191014" watchObservedRunningTime="2026-10-29 21:55:07.509488403 +0000 UTC m=+528.582445367"
Oct 29 21:55:07.553042 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:07.552990    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx" podStartSLOduration=2.552975606 podStartE2EDuration="2.552975606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:07.552841565 +0000 UTC m=+528.625798555" watchObservedRunningTime="2026-10-29 21:55:07.552975606 +0000 UTC m=+528.625932611"
Oct 29 21:55:09.721276 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:55:09.721236    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:55:09.942757 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:09.942729    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7"]
Oct 29 21:55:09.946822 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:09.946802    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7"
Oct 29 21:55:09.962729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:09.962707    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f97668b9\""
Oct 29 21:55:09.974396 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:09.974329    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f97668b9-dockercfg-mp8cj\""
Oct 29 21:55:09.976564 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:09.976543    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7"]
Oct 29 21:55:10.046308 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:10.044478    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f97668b9-65c448566d-bkh29"]
Oct 29 21:55:10.071351 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:10.058633    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfbgf\" (UniqueName: \"kubernetes.io/projected/7c51f1c6-c2da-492c-ae4c-11979c06e914-kube-api-access-rfbgf\") pod \"payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7\" (UID: \"7c51f1c6-c2da-492c-ae4c-11979c06e914\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7"
Oct 29 21:55:10.071351 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:10.058800    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c51f1c6-c2da-492c-ae4c-11979c06e914-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7\" (UID: \"7c51f1c6-c2da-492c-ae4c-11979c06e914\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7"
Oct 29 21:55:10.076967 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:10.071777    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f97668b9-65c448566d-bkh29"
Oct 29 21:55:10.085154 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:10.085117    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f97668b9-65c448566d-bkh29"]
Oct 29 21:55:10.160098 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:10.160065    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c51f1c6-c2da-492c-ae4c-11979c06e914-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7\" (UID: \"7c51f1c6-c2da-492c-ae4c-11979c06e914\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7"
Oct 29 21:55:10.160268 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:10.160132    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfbgf\" (UniqueName: \"kubernetes.io/projected/7c51f1c6-c2da-492c-ae4c-11979c06e914-kube-api-access-rfbgf\") pod \"payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7\" (UID: \"7c51f1c6-c2da-492c-ae4c-11979c06e914\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7"
Oct 29 21:55:10.161185 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:10.161153    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c51f1c6-c2da-492c-ae4c-11979c06e914-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7\" (UID: \"7c51f1c6-c2da-492c-ae4c-11979c06e914\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7"
Oct 29 21:55:10.183615 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:10.183593    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfbgf\" (UniqueName: \"kubernetes.io/projected/7c51f1c6-c2da-492c-ae4c-11979c06e914-kube-api-access-rfbgf\") pod \"payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7\" (UID: \"7c51f1c6-c2da-492c-ae4c-11979c06e914\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7"
Oct 29 21:55:10.261745 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:10.260520    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7"
Oct 29 21:55:10.261745 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:10.261089    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgzdb\" (UniqueName: \"kubernetes.io/projected/e3c6f723-516c-411d-88c4-798c15811838-kube-api-access-kgzdb\") pod \"payload-processing-e2e-ait-f97668b9-65c448566d-bkh29\" (UID: \"e3c6f723-516c-411d-88c4-798c15811838\") " pod="openshift-ingress/payload-processing-e2e-ait-f97668b9-65c448566d-bkh29"
Oct 29 21:55:10.261745 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:10.261214    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c6f723-516c-411d-88c4-798c15811838-plugins-config-volume\") pod \"payload-processing-e2e-ait-f97668b9-65c448566d-bkh29\" (UID: \"e3c6f723-516c-411d-88c4-798c15811838\") " pod="openshift-ingress/payload-processing-e2e-ait-f97668b9-65c448566d-bkh29"
Oct 29 21:55:10.363746 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:10.362668    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgzdb\" (UniqueName: \"kubernetes.io/projected/e3c6f723-516c-411d-88c4-798c15811838-kube-api-access-kgzdb\") pod \"payload-processing-e2e-ait-f97668b9-65c448566d-bkh29\" (UID: \"e3c6f723-516c-411d-88c4-798c15811838\") " pod="openshift-ingress/payload-processing-e2e-ait-f97668b9-65c448566d-bkh29"
Oct 29 21:55:10.363746 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:10.362750    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c6f723-516c-411d-88c4-798c15811838-plugins-config-volume\") pod \"payload-processing-e2e-ait-f97668b9-65c448566d-bkh29\" (UID: \"e3c6f723-516c-411d-88c4-798c15811838\") " pod="openshift-ingress/payload-processing-e2e-ait-f97668b9-65c448566d-bkh29"
Oct 29 21:55:10.363746 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:10.363673    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c6f723-516c-411d-88c4-798c15811838-plugins-config-volume\") pod \"payload-processing-e2e-ait-f97668b9-65c448566d-bkh29\" (UID: \"e3c6f723-516c-411d-88c4-798c15811838\") " pod="openshift-ingress/payload-processing-e2e-ait-f97668b9-65c448566d-bkh29"
Oct 29 21:55:10.388425 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:10.388389    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgzdb\" (UniqueName: \"kubernetes.io/projected/e3c6f723-516c-411d-88c4-798c15811838-kube-api-access-kgzdb\") pod \"payload-processing-e2e-ait-f97668b9-65c448566d-bkh29\" (UID: \"e3c6f723-516c-411d-88c4-798c15811838\") " pod="openshift-ingress/payload-processing-e2e-ait-f97668b9-65c448566d-bkh29"
Oct 29 21:55:10.410023 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:10.409998    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f97668b9-65c448566d-bkh29"
Oct 29 21:55:10.615955 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:10.615915    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7"]
Oct 29 21:55:10.616279 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:55:10.616249    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c51f1c6_c2da_492c_ae4c_11979c06e914.slice/crio-aeed3c4c6b7c14c9f2c044614259fb2428e60fb0069980e586890c5714932174 WatchSource:0}: Error finding container aeed3c4c6b7c14c9f2c044614259fb2428e60fb0069980e586890c5714932174: Status 404 returned error can't find the container with id aeed3c4c6b7c14c9f2c044614259fb2428e60fb0069980e586890c5714932174
Oct 29 21:55:10.713344 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:10.713315    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f97668b9-65c448566d-bkh29"]
Oct 29 21:55:10.715515 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:55:10.715482    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c6f723_516c_411d_88c4_798c15811838.slice/crio-35ad2c7fc6379bfed631c09c7d05572b96740dfa7a6ceba48bf046db34a35be2 WatchSource:0}: Error finding container 35ad2c7fc6379bfed631c09c7d05572b96740dfa7a6ceba48bf046db34a35be2: Status 404 returned error can't find the container with id 35ad2c7fc6379bfed631c09c7d05572b96740dfa7a6ceba48bf046db34a35be2
Oct 29 21:55:11.461048 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:11.461014    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr"
Oct 29 21:55:11.497851 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:11.497815    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7" event={"ID":"7c51f1c6-c2da-492c-ae4c-11979c06e914","Type":"ContainerStarted","Data":"200df1cd6835cf4612e1144f51213f126fb6b5676144e17629998d1d566ceb99"}
Oct 29 21:55:11.498098 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:11.498076    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7" event={"ID":"7c51f1c6-c2da-492c-ae4c-11979c06e914","Type":"ContainerStarted","Data":"aeed3c4c6b7c14c9f2c044614259fb2428e60fb0069980e586890c5714932174"}
Oct 29 21:55:11.503483 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:11.503457    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7"
Oct 29 21:55:11.505913 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:11.505888    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f97668b9-65c448566d-bkh29" event={"ID":"e3c6f723-516c-411d-88c4-798c15811838","Type":"ContainerStarted","Data":"0e313e7eeba2df02eed89859252ba3658f93085f3c1b1a6a00a3c4734cc750d4"}
Oct 29 21:55:11.506068 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:11.506054    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f97668b9-65c448566d-bkh29" event={"ID":"e3c6f723-516c-411d-88c4-798c15811838","Type":"ContainerStarted","Data":"35ad2c7fc6379bfed631c09c7d05572b96740dfa7a6ceba48bf046db34a35be2"}
Oct 29 21:55:11.506747 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:11.506728    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f97668b9-65c448566d-bkh29"
Oct 29 21:55:11.555623 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:11.555563    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7" podStartSLOduration=2.555545112 podStartE2EDuration="2.555545112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:11.553846333 +0000 UTC m=+532.626803318" watchObservedRunningTime="2026-10-29 21:55:11.555545112 +0000 UTC m=+532.628502077"
Oct 29 21:55:11.617320 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:11.595405    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f97668b9-65c448566d-bkh29" podStartSLOduration=1.5953861200000001 podStartE2EDuration="1.59538612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:11.594703976 +0000 UTC m=+532.667660944" watchObservedRunningTime="2026-10-29 21:55:11.59538612 +0000 UTC m=+532.668343084"
Oct 29 21:55:11.828431 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:11.828394    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks"]
Oct 29 21:55:11.832413 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:11.832390    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks"
Oct 29 21:55:11.851506 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:11.851383    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks"]
Oct 29 21:55:11.851506 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:11.851489    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-e4e24e\""
Oct 29 21:55:11.851679 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:11.851663    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-e4e24e-dockercfg-xtkzj\""
Oct 29 21:55:11.928322 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:11.926744    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76"]
Oct 29 21:55:11.935022 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:11.934431    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76"
Oct 29 21:55:11.936280 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:11.936253    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76"]
Oct 29 21:55:12.056496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:12.035261    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba71124e-36fa-41ea-bb8b-4f4375a1ac7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks\" (UID: \"ba71124e-36fa-41ea-bb8b-4f4375a1ac7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks"
Oct 29 21:55:12.056496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:12.035328    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmlrg\" (UniqueName: \"kubernetes.io/projected/fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d-kube-api-access-wmlrg\") pod \"payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76\" (UID: \"fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76"
Oct 29 21:55:12.056496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:12.035388    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhmnq\" (UniqueName: \"kubernetes.io/projected/ba71124e-36fa-41ea-bb8b-4f4375a1ac7f-kube-api-access-rhmnq\") pod \"payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks\" (UID: \"ba71124e-36fa-41ea-bb8b-4f4375a1ac7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks"
Oct 29 21:55:12.056496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:12.035452    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76\" (UID: \"fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76"
Oct 29 21:55:12.136325 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:12.136259    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76\" (UID: \"fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76"
Oct 29 21:55:12.136495 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:12.136403    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba71124e-36fa-41ea-bb8b-4f4375a1ac7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks\" (UID: \"ba71124e-36fa-41ea-bb8b-4f4375a1ac7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks"
Oct 29 21:55:12.136495 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:12.136453    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmlrg\" (UniqueName: \"kubernetes.io/projected/fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d-kube-api-access-wmlrg\") pod \"payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76\" (UID: \"fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76"
Oct 29 21:55:12.136634 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:12.136511    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhmnq\" (UniqueName: \"kubernetes.io/projected/ba71124e-36fa-41ea-bb8b-4f4375a1ac7f-kube-api-access-rhmnq\") pod \"payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks\" (UID: \"ba71124e-36fa-41ea-bb8b-4f4375a1ac7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks"
Oct 29 21:55:12.137567 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:12.137540    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76\" (UID: \"fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76"
Oct 29 21:55:12.137700 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:12.137627    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba71124e-36fa-41ea-bb8b-4f4375a1ac7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks\" (UID: \"ba71124e-36fa-41ea-bb8b-4f4375a1ac7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks"
Oct 29 21:55:12.162789 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:12.162758    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhmnq\" (UniqueName: \"kubernetes.io/projected/ba71124e-36fa-41ea-bb8b-4f4375a1ac7f-kube-api-access-rhmnq\") pod \"payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks\" (UID: \"ba71124e-36fa-41ea-bb8b-4f4375a1ac7f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks"
Oct 29 21:55:12.163935 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:12.163914    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmlrg\" (UniqueName: \"kubernetes.io/projected/fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d-kube-api-access-wmlrg\") pod \"payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76\" (UID: \"fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76"
Oct 29 21:55:12.258777 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:12.258729    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76"
Oct 29 21:55:12.447102 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:12.446851    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks"
Oct 29 21:55:12.625583 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:12.625411    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76"]
Oct 29 21:55:12.628094 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:55:12.628066    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc1ce1e_a65b_4e84_aa01_4dfc9780e14d.slice/crio-e058b8a951a50694a65a9519623693c90cc17bf0a97510461ba52d87d2863cf1 WatchSource:0}: Error finding container e058b8a951a50694a65a9519623693c90cc17bf0a97510461ba52d87d2863cf1: Status 404 returned error can't find the container with id e058b8a951a50694a65a9519623693c90cc17bf0a97510461ba52d87d2863cf1
Oct 29 21:55:12.917829 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:12.917795    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks"]
Oct 29 21:55:13.505316 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:13.504472    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4"
Oct 29 21:55:13.564248 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:13.564208    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks" event={"ID":"ba71124e-36fa-41ea-bb8b-4f4375a1ac7f","Type":"ContainerStarted","Data":"cfc9f9c09b2451d63e6b2b1c7c4651805da87a0ca9f8e8c94246f3f9ea1545d4"}
Oct 29 21:55:13.564425 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:13.564298    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks" event={"ID":"ba71124e-36fa-41ea-bb8b-4f4375a1ac7f","Type":"ContainerStarted","Data":"13c93476dfb3228bf138acaa627bf51c62090a58dd7639f268b67ab2eb96e733"}
Oct 29 21:55:13.564880 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:13.564860    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks"
Oct 29 21:55:13.566615 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:13.566586    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76" event={"ID":"fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d","Type":"ContainerStarted","Data":"12848ab4a37ecb12940a56ef55f814d03db31fd79418c0eef3ff0bc5f94373cd"}
Oct 29 21:55:13.566771 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:13.566631    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76" event={"ID":"fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d","Type":"ContainerStarted","Data":"e058b8a951a50694a65a9519623693c90cc17bf0a97510461ba52d87d2863cf1"}
Oct 29 21:55:13.567211 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:13.567192    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76"
Oct 29 21:55:13.632310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:13.627328    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks" podStartSLOduration=2.627308587 podStartE2EDuration="2.627308587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:13.626577709 +0000 UTC m=+534.699534673" watchObservedRunningTime="2026-10-29 21:55:13.627308587 +0000 UTC m=+534.700265572"
Oct 29 21:55:15.287309 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:15.285548    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76" podStartSLOduration=4.285527233 podStartE2EDuration="4.285527233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:13.68418289 +0000 UTC m=+534.757139855" watchObservedRunningTime="2026-10-29 21:55:15.285527233 +0000 UTC m=+536.358484195"
Oct 29 21:55:15.292306 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:15.288677    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2"]
Oct 29 21:55:15.297733 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:15.297711    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2"
Oct 29 21:55:15.330617 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:15.328479    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-8fba532a-serving-cert\""
Oct 29 21:55:15.333416 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:15.331832    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2"]
Oct 29 21:55:15.422058 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:15.421950    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-openshift-service-ca\") pod \"maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2\" (UID: \"0eb99ce9-22ff-4375-9d98-7cdb1a9675d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2"
Oct 29 21:55:15.422058 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:15.422010    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vn8x\" (UniqueName: \"kubernetes.io/projected/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-kube-api-access-6vn8x\") pod \"maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2\" (UID: \"0eb99ce9-22ff-4375-9d98-7cdb1a9675d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2"
Oct 29 21:55:15.422369 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:15.422117    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-metrics-certs\") pod \"maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2\" (UID: \"0eb99ce9-22ff-4375-9d98-7cdb1a9675d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2"
Oct 29 21:55:15.422369 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:15.422179    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-maas-api-tls\") pod \"maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2\" (UID: \"0eb99ce9-22ff-4375-9d98-7cdb1a9675d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2"
Oct 29 21:55:15.533158 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:15.533106    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-openshift-service-ca\") pod \"maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2\" (UID: \"0eb99ce9-22ff-4375-9d98-7cdb1a9675d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2"
Oct 29 21:55:15.534331 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:15.534305    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vn8x\" (UniqueName: \"kubernetes.io/projected/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-kube-api-access-6vn8x\") pod \"maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2\" (UID: \"0eb99ce9-22ff-4375-9d98-7cdb1a9675d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2"
Oct 29 21:55:15.534864 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:15.534218    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-openshift-service-ca\") pod \"maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2\" (UID: \"0eb99ce9-22ff-4375-9d98-7cdb1a9675d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2"
Oct 29 21:55:15.535009 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:15.534835    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-metrics-certs\") pod \"maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2\" (UID: \"0eb99ce9-22ff-4375-9d98-7cdb1a9675d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2"
Oct 29 21:55:15.535618 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:15.535601    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-maas-api-tls\") pod \"maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2\" (UID: \"0eb99ce9-22ff-4375-9d98-7cdb1a9675d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2"
Oct 29 21:55:15.557843 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:15.557778    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-maas-api-tls\") pod \"maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2\" (UID: \"0eb99ce9-22ff-4375-9d98-7cdb1a9675d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2"
Oct 29 21:55:15.557973 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:15.557854    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-metrics-certs\") pod \"maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2\" (UID: \"0eb99ce9-22ff-4375-9d98-7cdb1a9675d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2"
Oct 29 21:55:15.562309 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:15.560991    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vn8x\" (UniqueName: \"kubernetes.io/projected/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-kube-api-access-6vn8x\") pod \"maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2\" (UID: \"0eb99ce9-22ff-4375-9d98-7cdb1a9675d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2"
Oct 29 21:55:15.640597 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:15.640337    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2"
Oct 29 21:55:16.022355 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:16.020886    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2"]
Oct 29 21:55:16.458626 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:16.458607    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l"
Oct 29 21:55:16.458977 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:16.458664    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6"
Oct 29 21:55:16.611329 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:16.609482    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2" event={"ID":"0eb99ce9-22ff-4375-9d98-7cdb1a9675d8","Type":"ContainerStarted","Data":"c1d9f8375a0ccdc1a1e452f154d496dbdbc2b99ef45398318be05d4aef6beaa8"}
Oct 29 21:55:16.611329 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:16.609547    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2" event={"ID":"0eb99ce9-22ff-4375-9d98-7cdb1a9675d8","Type":"ContainerStarted","Data":"14dabf8472e923f938acc948a33f62e3b6815bcc24568e4ebe67a3c0efb3fb7d"}
Oct 29 21:55:16.611329 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:16.610417    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2"
Oct 29 21:55:16.653507 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:16.653408    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2" podStartSLOduration=1.255779363 podStartE2EDuration="1.653390823s" totalImagesPullingTime="397.61146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:16.652503492 +0000 UTC m=+537.725460473" watchObservedRunningTime="2026-10-29 21:55:16.653390823 +0000 UTC m=+537.726347788"
Oct 29 21:55:18.473228 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:18.473193    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx"
Oct 29 21:55:18.473762 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:18.473733    4962 patch_prober.go:29] interesting pod/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 21:55:18.474012 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:18.473812    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc" podUID="ac4fac5d-8644-4da7-82a0-aea07ee3f699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:55:20.636173 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:20.636088    4962 generic.go:356] "Generic (PLEG): container finished" podID="590636b8-d3a4-4c12-9c93-7755dbdf4f37" containerID="9cecdea73b7afddf02f82b32c4b7a3dceae967f59558bd45f1c9ba2b1ff8dc79" exitCode=0
Oct 29 21:55:20.636631 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:20.636168    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm" event={"ID":"590636b8-d3a4-4c12-9c93-7755dbdf4f37","Type":"ContainerDied","Data":"9cecdea73b7afddf02f82b32c4b7a3dceae967f59558bd45f1c9ba2b1ff8dc79"}
Oct 29 21:55:20.636705 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:20.636667    4962 scope.go:122] "RemoveContainer" containerID="9cecdea73b7afddf02f82b32c4b7a3dceae967f59558bd45f1c9ba2b1ff8dc79"
Oct 29 21:55:20.637930 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:20.637898    4962 generic.go:356] "Generic (PLEG): container finished" podID="ac4fac5d-8644-4da7-82a0-aea07ee3f699" containerID="1c1e0a166dd6749274c22ddd84d5179566f0977cbe88af58d7de628ce5d6b763" exitCode=0
Oct 29 21:55:20.638048 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:20.637969    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc" event={"ID":"ac4fac5d-8644-4da7-82a0-aea07ee3f699","Type":"ContainerDied","Data":"1c1e0a166dd6749274c22ddd84d5179566f0977cbe88af58d7de628ce5d6b763"}
Oct 29 21:55:20.638128 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:20.638100    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:55:20.638183 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:20.638171    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:55:20.638218 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:20.638206    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:55:20.638338 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:20.638325    4962 scope.go:122] "RemoveContainer" containerID="1c1e0a166dd6749274c22ddd84d5179566f0977cbe88af58d7de628ce5d6b763"
Oct 29 21:55:20.639511 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:20.639489    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:55:20.639630 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:20.639547    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:55:20.639630 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:20.639581    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:55:21.645326 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:21.645281    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm" event={"ID":"590636b8-d3a4-4c12-9c93-7755dbdf4f37","Type":"ContainerStarted","Data":"1a5b3808bb616bccbecdbaf20f8ea9049d7e921f19f465033c6ef3cf25f3cae7"}
Oct 29 21:55:21.647256 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:21.647234    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc" event={"ID":"ac4fac5d-8644-4da7-82a0-aea07ee3f699","Type":"ContainerStarted","Data":"c9475af40eca959c9e42ac9ede5f83879427c452c4486b1ddbaf29b012046677"}
Oct 29 21:55:22.312310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:22.311683    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:55:22.312310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:22.311808    4962 patch_prober.go:29] interesting pod/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 29 21:55:22.312310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:22.311864    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm" podUID="590636b8-d3a4-4c12-9c93-7755dbdf4f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 29 21:55:22.591367 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:22.590654    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:55:22.591367 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:22.590769    4962 patch_prober.go:29] interesting pod/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 29 21:55:22.591367 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:22.590822    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc" podUID="ac4fac5d-8644-4da7-82a0-aea07ee3f699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 29 21:55:23.309310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:23.308662    4962 patch_prober.go:29] interesting pod/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 29 21:55:23.309310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:23.308735    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm" podUID="590636b8-d3a4-4c12-9c93-7755dbdf4f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 29 21:55:23.570010 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:23.569931    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f97668b9-65c448566d-bkh29"
Oct 29 21:55:23.570010 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:23.569996    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7"
Oct 29 21:55:23.593310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:23.593066    4962 patch_prober.go:29] interesting pod/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 29 21:55:23.593310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:23.593127    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc" podUID="ac4fac5d-8644-4da7-82a0-aea07ee3f699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 29 21:55:23.629092 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:23.628515    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2"
Oct 29 21:55:24.311309 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:24.310737    4962 patch_prober.go:29] interesting pod/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 29 21:55:24.311309 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:24.310798    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm" podUID="590636b8-d3a4-4c12-9c93-7755dbdf4f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 29 21:55:24.593364 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:24.592678    4962 patch_prober.go:29] interesting pod/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 29 21:55:24.593364 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:24.592753    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc" podUID="ac4fac5d-8644-4da7-82a0-aea07ee3f699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 29 21:55:24.619312 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:24.616576    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76"
Oct 29 21:55:25.319312 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:25.311690    4962 patch_prober.go:29] interesting pod/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 29 21:55:25.319312 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:25.311755    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm" podUID="590636b8-d3a4-4c12-9c93-7755dbdf4f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 29 21:55:25.596366 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:25.595682    4962 patch_prober.go:29] interesting pod/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 29 21:55:25.596366 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:25.595742    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc" podUID="ac4fac5d-8644-4da7-82a0-aea07ee3f699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 29 21:55:25.625481 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:25.625451    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks"
Oct 29 21:55:26.226366 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:26.226327    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf"]
Oct 29 21:55:26.231741 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:26.231717    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf"
Oct 29 21:55:26.251421 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:26.251396    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-a3c02e-serving-cert\""
Oct 29 21:55:26.312331 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:26.310679    4962 patch_prober.go:29] interesting pod/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body=
Oct 29 21:55:26.312331 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:26.310739    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm" podUID="590636b8-d3a4-4c12-9c93-7755dbdf4f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused"
Oct 29 21:55:26.432351 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:26.396264    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf"]
Oct 29 21:55:26.432955 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:26.399947    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cx95\" (UniqueName: \"kubernetes.io/projected/7cb302c3-d428-4a64-a44e-d5661108233d-kube-api-access-5cx95\") pod \"maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf\" (UID: \"7cb302c3-d428-4a64-a44e-d5661108233d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf"
Oct 29 21:55:26.433161 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:26.433141    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cb302c3-d428-4a64-a44e-d5661108233d-metrics-certs\") pod \"maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf\" (UID: \"7cb302c3-d428-4a64-a44e-d5661108233d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf"
Oct 29 21:55:26.433410 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:26.433387    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb302c3-d428-4a64-a44e-d5661108233d-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf\" (UID: \"7cb302c3-d428-4a64-a44e-d5661108233d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf"
Oct 29 21:55:26.433603 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:26.433585    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cb302c3-d428-4a64-a44e-d5661108233d-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf\" (UID: \"7cb302c3-d428-4a64-a44e-d5661108233d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf"
Oct 29 21:55:26.561406 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:26.560714    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cb302c3-d428-4a64-a44e-d5661108233d-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf\" (UID: \"7cb302c3-d428-4a64-a44e-d5661108233d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf"
Oct 29 21:55:26.561406 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:26.560824    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cx95\" (UniqueName: \"kubernetes.io/projected/7cb302c3-d428-4a64-a44e-d5661108233d-kube-api-access-5cx95\") pod \"maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf\" (UID: \"7cb302c3-d428-4a64-a44e-d5661108233d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf"
Oct 29 21:55:26.561406 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:26.560852    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cb302c3-d428-4a64-a44e-d5661108233d-metrics-certs\") pod \"maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf\" (UID: \"7cb302c3-d428-4a64-a44e-d5661108233d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf"
Oct 29 21:55:26.561406 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:26.560875    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb302c3-d428-4a64-a44e-d5661108233d-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf\" (UID: \"7cb302c3-d428-4a64-a44e-d5661108233d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf"
Oct 29 21:55:26.561734 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:26.561530    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb302c3-d428-4a64-a44e-d5661108233d-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf\" (UID: \"7cb302c3-d428-4a64-a44e-d5661108233d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf"
Oct 29 21:55:26.569404 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:26.569331    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cb302c3-d428-4a64-a44e-d5661108233d-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf\" (UID: \"7cb302c3-d428-4a64-a44e-d5661108233d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf"
Oct 29 21:55:26.573308 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:26.570125    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cb302c3-d428-4a64-a44e-d5661108233d-metrics-certs\") pod \"maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf\" (UID: \"7cb302c3-d428-4a64-a44e-d5661108233d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf"
Oct 29 21:55:26.591262 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:26.591233    4962 patch_prober.go:29] interesting pod/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body=
Oct 29 21:55:26.591547 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:26.591521    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc" podUID="ac4fac5d-8644-4da7-82a0-aea07ee3f699" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused"
Oct 29 21:55:26.599002 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:26.598943    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cx95\" (UniqueName: \"kubernetes.io/projected/7cb302c3-d428-4a64-a44e-d5661108233d-kube-api-access-5cx95\") pod \"maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf\" (UID: \"7cb302c3-d428-4a64-a44e-d5661108233d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf"
Oct 29 21:55:26.844988 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:26.844898    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf"
Oct 29 21:55:27.176239 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:27.176085    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf"]
Oct 29 21:55:27.179174 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:55:27.179141    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb302c3_d428_4a64_a44e_d5661108233d.slice/crio-c1600b8900822044a95043d99ce407d1bbabcf2acbbc91dd751eead5022ec582 WatchSource:0}: Error finding container c1600b8900822044a95043d99ce407d1bbabcf2acbbc91dd751eead5022ec582: Status 404 returned error can't find the container with id c1600b8900822044a95043d99ce407d1bbabcf2acbbc91dd751eead5022ec582
Oct 29 21:55:27.417943 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:27.417910    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:55:27.418557 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:27.418535    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:55:27.421420 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:27.421399    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:55:27.593860 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:27.593831    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:55:27.594215 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:27.594116    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:55:27.594734 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:27.594718    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:55:27.742012 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:27.741929    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf" event={"ID":"7cb302c3-d428-4a64-a44e-d5661108233d","Type":"ContainerStarted","Data":"5834de82e79b26260dcc022d78a1e49b362baa587ea17e68f6b5fbb609230257"}
Oct 29 21:55:27.742012 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:27.742001    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf" event={"ID":"7cb302c3-d428-4a64-a44e-d5661108233d","Type":"ContainerStarted","Data":"c1600b8900822044a95043d99ce407d1bbabcf2acbbc91dd751eead5022ec582"}
Oct 29 21:55:27.742225 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:27.742205    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf"
Oct 29 21:55:27.790134 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:27.790070    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf" podStartSLOduration=1.489635038 podStartE2EDuration="1.790051047s" totalImagesPullingTime="300.416009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:27.78991548 +0000 UTC m=+548.862872443" watchObservedRunningTime="2026-10-29 21:55:27.790051047 +0000 UTC m=+548.863008012"
Oct 29 21:55:28.606827 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:28.606788    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6"]
Oct 29 21:55:28.636153 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:28.636123    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l"]
Oct 29 21:55:32.897550 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:32.891151    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx"]
Oct 29 21:55:33.134844 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.134808    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4"]
Oct 29 21:55:33.135097 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.135052    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4" podUID="d62760e8-caa9-49a1-976d-c56196ecd567" containerName="maas-api" containerID="cri-o://9ca4cb84c2aa6cb553fe1e30a966dc8b56c4f766e05c5cc18d18ff8713e93e13" gracePeriod=30
Oct 29 21:55:33.426564 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.426540    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4"
Oct 29 21:55:33.458902 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.458879    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d62760e8-caa9-49a1-976d-c56196ecd567-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d62760e8-caa9-49a1-976d-c56196ecd567-maas-api-tls\") pod \"d62760e8-caa9-49a1-976d-c56196ecd567\" (UID: \"d62760e8-caa9-49a1-976d-c56196ecd567\") "
Oct 29 21:55:33.459017 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.458915    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d62760e8-caa9-49a1-976d-c56196ecd567-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62760e8-caa9-49a1-976d-c56196ecd567-openshift-service-ca\") pod \"d62760e8-caa9-49a1-976d-c56196ecd567\" (UID: \"d62760e8-caa9-49a1-976d-c56196ecd567\") "
Oct 29 21:55:33.459017 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.458953    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d62760e8-caa9-49a1-976d-c56196ecd567-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d62760e8-caa9-49a1-976d-c56196ecd567-metrics-certs\") pod \"d62760e8-caa9-49a1-976d-c56196ecd567\" (UID: \"d62760e8-caa9-49a1-976d-c56196ecd567\") "
Oct 29 21:55:33.459137 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.459043    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d62760e8-caa9-49a1-976d-c56196ecd567-kube-api-access-mbxcc\" (UniqueName: \"kubernetes.io/projected/d62760e8-caa9-49a1-976d-c56196ecd567-kube-api-access-mbxcc\") pod \"d62760e8-caa9-49a1-976d-c56196ecd567\" (UID: \"d62760e8-caa9-49a1-976d-c56196ecd567\") "
Oct 29 21:55:33.459363 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.459337    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d62760e8-caa9-49a1-976d-c56196ecd567-openshift-service-ca" pod "d62760e8-caa9-49a1-976d-c56196ecd567" (UID: "d62760e8-caa9-49a1-976d-c56196ecd567"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:33.460959 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.460936    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d62760e8-caa9-49a1-976d-c56196ecd567-maas-api-tls" pod "d62760e8-caa9-49a1-976d-c56196ecd567" (UID: "d62760e8-caa9-49a1-976d-c56196ecd567"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:55:33.461042 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.461009    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d62760e8-caa9-49a1-976d-c56196ecd567-metrics-certs" pod "d62760e8-caa9-49a1-976d-c56196ecd567" (UID: "d62760e8-caa9-49a1-976d-c56196ecd567"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:55:33.461154 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.461137    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d62760e8-caa9-49a1-976d-c56196ecd567-kube-api-access-mbxcc" pod "d62760e8-caa9-49a1-976d-c56196ecd567" (UID: "d62760e8-caa9-49a1-976d-c56196ecd567"). InnerVolumeSpecName "kube-api-access-mbxcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:33.564669 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.563363    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbxcc\" (UniqueName: \"kubernetes.io/projected/d62760e8-caa9-49a1-976d-c56196ecd567-kube-api-access-mbxcc\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:33.564669 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.563414    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d62760e8-caa9-49a1-976d-c56196ecd567-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:33.564669 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.563430    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d62760e8-caa9-49a1-976d-c56196ecd567-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:33.564669 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.563444    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d62760e8-caa9-49a1-976d-c56196ecd567-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:33.622322 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.614770    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6" podUID="ad9d9073-c42c-4c70-90a5-5971211c41b2" containerName="payload-processing" containerID="cri-o://6dd017dcc5533d9924ef61dbec91c12c1d9696dd6db7e853ee79cd0a1534a599" gracePeriod=25
Oct 29 21:55:33.637431 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.637399    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l" podUID="eb1dcd64-e3c0-4ac0-8941-c0845dba1979" containerName="payload-pre-processing" containerID="cri-o://fd898439e4264c4f0d52994ddab3a3e7562d846b96fa30993cf37f092e3a08d7" gracePeriod=25
Oct 29 21:55:33.758610 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.758582    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf"
Oct 29 21:55:33.810313 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.810220    4962 generic.go:356] "Generic (PLEG): container finished" podID="eb1dcd64-e3c0-4ac0-8941-c0845dba1979" containerID="fd898439e4264c4f0d52994ddab3a3e7562d846b96fa30993cf37f092e3a08d7" exitCode=0
Oct 29 21:55:33.810313 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.810307    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l" event={"ID":"eb1dcd64-e3c0-4ac0-8941-c0845dba1979","Type":"ContainerDied","Data":"fd898439e4264c4f0d52994ddab3a3e7562d846b96fa30993cf37f092e3a08d7"}
Oct 29 21:55:33.812544 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.811968    4962 generic.go:356] "Generic (PLEG): container finished" podID="ad9d9073-c42c-4c70-90a5-5971211c41b2" containerID="6dd017dcc5533d9924ef61dbec91c12c1d9696dd6db7e853ee79cd0a1534a599" exitCode=0
Oct 29 21:55:33.812544 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.812080    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6" event={"ID":"ad9d9073-c42c-4c70-90a5-5971211c41b2","Type":"ContainerDied","Data":"6dd017dcc5533d9924ef61dbec91c12c1d9696dd6db7e853ee79cd0a1534a599"}
Oct 29 21:55:33.814096 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.813740    4962 generic.go:356] "Generic (PLEG): container finished" podID="d62760e8-caa9-49a1-976d-c56196ecd567" containerID="9ca4cb84c2aa6cb553fe1e30a966dc8b56c4f766e05c5cc18d18ff8713e93e13" exitCode=0
Oct 29 21:55:33.814096 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.813799    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4" event={"ID":"d62760e8-caa9-49a1-976d-c56196ecd567","Type":"ContainerDied","Data":"9ca4cb84c2aa6cb553fe1e30a966dc8b56c4f766e05c5cc18d18ff8713e93e13"}
Oct 29 21:55:33.814096 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.813857    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4" event={"ID":"d62760e8-caa9-49a1-976d-c56196ecd567","Type":"ContainerDied","Data":"88cdc61c874f1875bb915ffcc84282d53ede6dff3489f76c097d07c5d6cc8ed1"}
Oct 29 21:55:33.814096 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.813886    4962 scope.go:122] "RemoveContainer" containerID="9ca4cb84c2aa6cb553fe1e30a966dc8b56c4f766e05c5cc18d18ff8713e93e13"
Oct 29 21:55:33.814388 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.814121    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4"
Oct 29 21:55:33.829264 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.827899    4962 scope.go:122] "RemoveContainer" containerID="9ca4cb84c2aa6cb553fe1e30a966dc8b56c4f766e05c5cc18d18ff8713e93e13"
Oct 29 21:55:33.829264 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:55:33.828176    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ca4cb84c2aa6cb553fe1e30a966dc8b56c4f766e05c5cc18d18ff8713e93e13\": container with ID starting with 9ca4cb84c2aa6cb553fe1e30a966dc8b56c4f766e05c5cc18d18ff8713e93e13 not found: ID does not exist" containerID="9ca4cb84c2aa6cb553fe1e30a966dc8b56c4f766e05c5cc18d18ff8713e93e13"
Oct 29 21:55:33.829264 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.828206    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca4cb84c2aa6cb553fe1e30a966dc8b56c4f766e05c5cc18d18ff8713e93e13"} err="failed to get container status \"9ca4cb84c2aa6cb553fe1e30a966dc8b56c4f766e05c5cc18d18ff8713e93e13\": rpc error: code = NotFound desc = could not find container \"9ca4cb84c2aa6cb553fe1e30a966dc8b56c4f766e05c5cc18d18ff8713e93e13\": container with ID starting with 9ca4cb84c2aa6cb553fe1e30a966dc8b56c4f766e05c5cc18d18ff8713e93e13 not found: ID does not exist"
Oct 29 21:55:33.869039 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.869008    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4"]
Oct 29 21:55:33.900313 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:33.898630    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64505fc5-7c9c88574d-s6tf4"]
Oct 29 21:55:34.017133 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:34.017092    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6"
Oct 29 21:55:34.045165 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:34.043144    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l"
Oct 29 21:55:34.092310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:34.071555    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb1dcd64-e3c0-4ac0-8941-c0845dba1979-kube-api-access-qlhw8\" (UniqueName: \"kubernetes.io/projected/eb1dcd64-e3c0-4ac0-8941-c0845dba1979-kube-api-access-qlhw8\") pod \"eb1dcd64-e3c0-4ac0-8941-c0845dba1979\" (UID: \"eb1dcd64-e3c0-4ac0-8941-c0845dba1979\") "
Oct 29 21:55:34.092310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:34.071612    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad9d9073-c42c-4c70-90a5-5971211c41b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad9d9073-c42c-4c70-90a5-5971211c41b2-plugins-config-volume\") pod \"ad9d9073-c42c-4c70-90a5-5971211c41b2\" (UID: \"ad9d9073-c42c-4c70-90a5-5971211c41b2\") "
Oct 29 21:55:34.092310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:34.071656    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad9d9073-c42c-4c70-90a5-5971211c41b2-kube-api-access-xp5xn\" (UniqueName: \"kubernetes.io/projected/ad9d9073-c42c-4c70-90a5-5971211c41b2-kube-api-access-xp5xn\") pod \"ad9d9073-c42c-4c70-90a5-5971211c41b2\" (UID: \"ad9d9073-c42c-4c70-90a5-5971211c41b2\") "
Oct 29 21:55:34.092310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:34.071796    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb1dcd64-e3c0-4ac0-8941-c0845dba1979-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb1dcd64-e3c0-4ac0-8941-c0845dba1979-plugins-config-volume\") pod \"eb1dcd64-e3c0-4ac0-8941-c0845dba1979\" (UID: \"eb1dcd64-e3c0-4ac0-8941-c0845dba1979\") "
Oct 29 21:55:34.098551 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:34.098516    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad9d9073-c42c-4c70-90a5-5971211c41b2-plugins-config-volume" pod "ad9d9073-c42c-4c70-90a5-5971211c41b2" (UID: "ad9d9073-c42c-4c70-90a5-5971211c41b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:34.098681 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:34.098520    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb1dcd64-e3c0-4ac0-8941-c0845dba1979-plugins-config-volume" pod "eb1dcd64-e3c0-4ac0-8941-c0845dba1979" (UID: "eb1dcd64-e3c0-4ac0-8941-c0845dba1979"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:34.100512 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:34.100470    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1dcd64-e3c0-4ac0-8941-c0845dba1979-kube-api-access-qlhw8" pod "eb1dcd64-e3c0-4ac0-8941-c0845dba1979" (UID: "eb1dcd64-e3c0-4ac0-8941-c0845dba1979"). InnerVolumeSpecName "kube-api-access-qlhw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:34.101191 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:34.101153    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad9d9073-c42c-4c70-90a5-5971211c41b2-kube-api-access-xp5xn" pod "ad9d9073-c42c-4c70-90a5-5971211c41b2" (UID: "ad9d9073-c42c-4c70-90a5-5971211c41b2"). InnerVolumeSpecName "kube-api-access-xp5xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:34.196321 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:34.195965    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb1dcd64-e3c0-4ac0-8941-c0845dba1979-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:34.196321 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:34.196004    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlhw8\" (UniqueName: \"kubernetes.io/projected/eb1dcd64-e3c0-4ac0-8941-c0845dba1979-kube-api-access-qlhw8\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:34.196321 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:34.196020    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad9d9073-c42c-4c70-90a5-5971211c41b2-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:34.196321 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:34.196033    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp5xn\" (UniqueName: \"kubernetes.io/projected/ad9d9073-c42c-4c70-90a5-5971211c41b2-kube-api-access-xp5xn\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:34.820098 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:34.820061    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6"
Oct 29 21:55:34.820098 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:34.820081    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6" event={"ID":"ad9d9073-c42c-4c70-90a5-5971211c41b2","Type":"ContainerDied","Data":"9e7bed16128698ae1f5bb0345feaf5ef7f54e020c75ab7f27814f37197ab97dc"}
Oct 29 21:55:34.820347 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:34.820183    4962 scope.go:122] "RemoveContainer" containerID="6dd017dcc5533d9924ef61dbec91c12c1d9696dd6db7e853ee79cd0a1534a599"
Oct 29 21:55:34.822565 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:34.822526    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l" event={"ID":"eb1dcd64-e3c0-4ac0-8941-c0845dba1979","Type":"ContainerDied","Data":"7d5ff22c954bee2dcbf1c3dc248068919428da253f6ca257bdb0ef6f37d1854b"}
Oct 29 21:55:34.822565 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:34.822549    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l"
Oct 29 21:55:34.830263 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:34.830244    4962 scope.go:122] "RemoveContainer" containerID="fd898439e4264c4f0d52994ddab3a3e7562d846b96fa30993cf37f092e3a08d7"
Oct 29 21:55:34.870769 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:34.870745    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6"]
Oct 29 21:55:34.897673 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:34.897648    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f5bae148-885f5c686-wfnp6"]
Oct 29 21:55:34.932123 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:34.932102    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l"]
Oct 29 21:55:34.937837 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:34.937815    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f5bae148-789f65679-l6x4l"]
Oct 29 21:55:35.058207 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:35.058179    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad9d9073-c42c-4c70-90a5-5971211c41b2" path="/var/lib/kubelet/pods/ad9d9073-c42c-4c70-90a5-5971211c41b2/volumes"
Oct 29 21:55:35.058631 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:35.058617    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d62760e8-caa9-49a1-976d-c56196ecd567" path="/var/lib/kubelet/pods/d62760e8-caa9-49a1-976d-c56196ecd567/volumes"
Oct 29 21:55:35.058950 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:35.058938    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb1dcd64-e3c0-4ac0-8941-c0845dba1979" path="/var/lib/kubelet/pods/eb1dcd64-e3c0-4ac0-8941-c0845dba1979/volumes"
Oct 29 21:55:35.782863 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:35.782789    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"]
Oct 29 21:55:35.783054 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:35.783030    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc" podUID="ac4fac5d-8644-4da7-82a0-aea07ee3f699" containerName="istio-proxy" containerID="cri-o://c9475af40eca959c9e42ac9ede5f83879427c452c4486b1ddbaf29b012046677" gracePeriod=30
Oct 29 21:55:37.897646 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:37.897602    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx" podUID="0a47e418-7c3c-4f35-a423-ac4296c9c8a7" containerName="payload-processing" containerID="cri-o://96e15802c93aa45b0470df239550af358d9c3a013ade0f52a51d6b35fecebf0a" gracePeriod=25
Oct 29 21:55:38.145155 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:38.145129    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx"
Oct 29 21:55:38.268746 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:38.268714    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a47e418-7c3c-4f35-a423-ac4296c9c8a7-kube-api-access-txxbr\" (UniqueName: \"kubernetes.io/projected/0a47e418-7c3c-4f35-a423-ac4296c9c8a7-kube-api-access-txxbr\") pod \"0a47e418-7c3c-4f35-a423-ac4296c9c8a7\" (UID: \"0a47e418-7c3c-4f35-a423-ac4296c9c8a7\") "
Oct 29 21:55:38.268939 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:38.268878    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a47e418-7c3c-4f35-a423-ac4296c9c8a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a47e418-7c3c-4f35-a423-ac4296c9c8a7-plugins-config-volume\") pod \"0a47e418-7c3c-4f35-a423-ac4296c9c8a7\" (UID: \"0a47e418-7c3c-4f35-a423-ac4296c9c8a7\") "
Oct 29 21:55:38.269339 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:38.269314    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a47e418-7c3c-4f35-a423-ac4296c9c8a7-plugins-config-volume" pod "0a47e418-7c3c-4f35-a423-ac4296c9c8a7" (UID: "0a47e418-7c3c-4f35-a423-ac4296c9c8a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:38.271266 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:38.271236    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a47e418-7c3c-4f35-a423-ac4296c9c8a7-kube-api-access-txxbr" pod "0a47e418-7c3c-4f35-a423-ac4296c9c8a7" (UID: "0a47e418-7c3c-4f35-a423-ac4296c9c8a7"). InnerVolumeSpecName "kube-api-access-txxbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:38.370315 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:38.370277    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txxbr\" (UniqueName: \"kubernetes.io/projected/0a47e418-7c3c-4f35-a423-ac4296c9c8a7-kube-api-access-txxbr\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:38.370315 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:38.370316    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a47e418-7c3c-4f35-a423-ac4296c9c8a7-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:38.826432 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:38.826398    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f97668b9-65c448566d-bkh29"]
Oct 29 21:55:38.851991 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:38.851960    4962 generic.go:356] "Generic (PLEG): container finished" podID="0a47e418-7c3c-4f35-a423-ac4296c9c8a7" containerID="96e15802c93aa45b0470df239550af358d9c3a013ade0f52a51d6b35fecebf0a" exitCode=0
Oct 29 21:55:38.852145 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:38.852010    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx"
Oct 29 21:55:38.852145 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:38.852050    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx" event={"ID":"0a47e418-7c3c-4f35-a423-ac4296c9c8a7","Type":"ContainerDied","Data":"96e15802c93aa45b0470df239550af358d9c3a013ade0f52a51d6b35fecebf0a"}
Oct 29 21:55:38.852145 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:38.852104    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx" event={"ID":"0a47e418-7c3c-4f35-a423-ac4296c9c8a7","Type":"ContainerDied","Data":"9e4d5e2fcf3e2fff44e46783440e65dd491066555e8c78df745b115ce1442976"}
Oct 29 21:55:38.852145 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:38.852126    4962 scope.go:122] "RemoveContainer" containerID="96e15802c93aa45b0470df239550af358d9c3a013ade0f52a51d6b35fecebf0a"
Oct 29 21:55:38.862443 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:38.862422    4962 scope.go:122] "RemoveContainer" containerID="96e15802c93aa45b0470df239550af358d9c3a013ade0f52a51d6b35fecebf0a"
Oct 29 21:55:38.862709 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:55:38.862688    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96e15802c93aa45b0470df239550af358d9c3a013ade0f52a51d6b35fecebf0a\": container with ID starting with 96e15802c93aa45b0470df239550af358d9c3a013ade0f52a51d6b35fecebf0a not found: ID does not exist" containerID="96e15802c93aa45b0470df239550af358d9c3a013ade0f52a51d6b35fecebf0a"
Oct 29 21:55:38.862773 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:38.862717    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e15802c93aa45b0470df239550af358d9c3a013ade0f52a51d6b35fecebf0a"} err="failed to get container status \"96e15802c93aa45b0470df239550af358d9c3a013ade0f52a51d6b35fecebf0a\": rpc error: code = NotFound desc = could not find container \"96e15802c93aa45b0470df239550af358d9c3a013ade0f52a51d6b35fecebf0a\": container with ID starting with 96e15802c93aa45b0470df239550af358d9c3a013ade0f52a51d6b35fecebf0a not found: ID does not exist"
Oct 29 21:55:38.864548 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:38.864528    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7"]
Oct 29 21:55:38.953454 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:38.953421    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx"]
Oct 29 21:55:38.963786 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:38.963759    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-64505fc5-77dc98d8c6-cwbwx"]
Oct 29 21:55:39.084828 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:39.084432    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a47e418-7c3c-4f35-a423-ac4296c9c8a7" path="/var/lib/kubelet/pods/0a47e418-7c3c-4f35-a423-ac4296c9c8a7/volumes"
Oct 29 21:55:39.727475 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:55:39.727437    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:55:39.864780 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:39.864740    4962 generic.go:356] "Generic (PLEG): container finished" podID="10a9a83c-2b74-4cba-8011-63a7f4f2f134" containerID="42d21d24e8726835a75e170c7a68ea46b03430e7e24ddfe6a2fce7c459e71fa8" exitCode=0
Oct 29 21:55:39.864996 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:39.864891    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh" event={"ID":"10a9a83c-2b74-4cba-8011-63a7f4f2f134","Type":"ContainerDied","Data":"42d21d24e8726835a75e170c7a68ea46b03430e7e24ddfe6a2fce7c459e71fa8"}
Oct 29 21:55:39.865514 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:39.865485    4962 scope.go:122] "RemoveContainer" containerID="42d21d24e8726835a75e170c7a68ea46b03430e7e24ddfe6a2fce7c459e71fa8"
Oct 29 21:55:39.868011 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:39.867830    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:55:39.868011 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:39.867901    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:55:39.868011 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:39.867938    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:55:40.604844 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:40.604615    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr"]
Oct 29 21:55:40.605344 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:40.604840    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr" podUID="d91e7562-34b9-4350-896a-c95260f5c07a" containerName="maas-api" containerID="cri-o://ae2f1fbec7120f85dfe25017f780112ab7486fab839439328db484933c58bceb" gracePeriod=30
Oct 29 21:55:40.940560 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:40.940092    4962 generic.go:356] "Generic (PLEG): container finished" podID="d91e7562-34b9-4350-896a-c95260f5c07a" containerID="ae2f1fbec7120f85dfe25017f780112ab7486fab839439328db484933c58bceb" exitCode=0
Oct 29 21:55:40.940560 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:40.940159    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr" event={"ID":"d91e7562-34b9-4350-896a-c95260f5c07a","Type":"ContainerDied","Data":"ae2f1fbec7120f85dfe25017f780112ab7486fab839439328db484933c58bceb"}
Oct 29 21:55:40.985766 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:40.983884    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh" event={"ID":"10a9a83c-2b74-4cba-8011-63a7f4f2f134","Type":"ContainerStarted","Data":"b6d825ce4356eeb4e2ff4b66ef44bf6d0e0147ac6e833ad0797b3d49dec9e487"}
Oct 29 21:55:41.037532 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.037226    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr"
Oct 29 21:55:41.109038 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.109004    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:55:41.109405 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.109376    4962 patch_prober.go:29] interesting pod/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 29 21:55:41.109576 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.109549    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh" podUID="10a9a83c-2b74-4cba-8011-63a7f4f2f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 29 21:55:41.121604 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.121584    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d91e7562-34b9-4350-896a-c95260f5c07a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91e7562-34b9-4350-896a-c95260f5c07a-metrics-certs\") pod \"d91e7562-34b9-4350-896a-c95260f5c07a\" (UID: \"d91e7562-34b9-4350-896a-c95260f5c07a\") "
Oct 29 21:55:41.121798 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.121780    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91e7562-34b9-4350-896a-c95260f5c07a-kube-api-access-z5z26\" (UniqueName: \"kubernetes.io/projected/d91e7562-34b9-4350-896a-c95260f5c07a-kube-api-access-z5z26\") pod \"d91e7562-34b9-4350-896a-c95260f5c07a\" (UID: \"d91e7562-34b9-4350-896a-c95260f5c07a\") "
Oct 29 21:55:41.121914 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.121862    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d91e7562-34b9-4350-896a-c95260f5c07a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d91e7562-34b9-4350-896a-c95260f5c07a-openshift-service-ca\") pod \"d91e7562-34b9-4350-896a-c95260f5c07a\" (UID: \"d91e7562-34b9-4350-896a-c95260f5c07a\") "
Oct 29 21:55:41.122075 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.122061    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d91e7562-34b9-4350-896a-c95260f5c07a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d91e7562-34b9-4350-896a-c95260f5c07a-maas-api-tls\") pod \"d91e7562-34b9-4350-896a-c95260f5c07a\" (UID: \"d91e7562-34b9-4350-896a-c95260f5c07a\") "
Oct 29 21:55:41.122506 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.122473    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d91e7562-34b9-4350-896a-c95260f5c07a-openshift-service-ca" pod "d91e7562-34b9-4350-896a-c95260f5c07a" (UID: "d91e7562-34b9-4350-896a-c95260f5c07a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:41.125318 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.124828    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d91e7562-34b9-4350-896a-c95260f5c07a-maas-api-tls" pod "d91e7562-34b9-4350-896a-c95260f5c07a" (UID: "d91e7562-34b9-4350-896a-c95260f5c07a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:55:41.125318 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.124928    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91e7562-34b9-4350-896a-c95260f5c07a-kube-api-access-z5z26" pod "d91e7562-34b9-4350-896a-c95260f5c07a" (UID: "d91e7562-34b9-4350-896a-c95260f5c07a"). InnerVolumeSpecName "kube-api-access-z5z26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:41.125318 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.124957    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d91e7562-34b9-4350-896a-c95260f5c07a-metrics-certs" pod "d91e7562-34b9-4350-896a-c95260f5c07a" (UID: "d91e7562-34b9-4350-896a-c95260f5c07a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:55:41.223779 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.223691    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d91e7562-34b9-4350-896a-c95260f5c07a-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:41.223779 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.223729    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5z26\" (UniqueName: \"kubernetes.io/projected/d91e7562-34b9-4350-896a-c95260f5c07a-kube-api-access-z5z26\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:41.223779 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.223744    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d91e7562-34b9-4350-896a-c95260f5c07a-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:41.223779 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.223758    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d91e7562-34b9-4350-896a-c95260f5c07a-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:41.246358 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.246336    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:55:41.425902 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.425719    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-data\") pod \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") "
Oct 29 21:55:41.425902 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.425820    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac4fac5d-8644-4da7-82a0-aea07ee3f699-kube-api-access-xfx9w\" (UniqueName: \"kubernetes.io/projected/ac4fac5d-8644-4da7-82a0-aea07ee3f699-kube-api-access-xfx9w\") pod \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") "
Oct 29 21:55:41.425902 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.425883    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-token\" (UniqueName: \"kubernetes.io/projected/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-token\") pod \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") "
Oct 29 21:55:41.426221 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.425969    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-workload-certs\") pod \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") "
Oct 29 21:55:41.426221 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.426014    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-envoy\") pod \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") "
Oct 29 21:55:41.426221 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.426042    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-workload-socket\") pod \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") "
Oct 29 21:55:41.426221 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.426096    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-podinfo\") pod \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") "
Oct 29 21:55:41.426221 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.426135    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-credential-socket\") pod \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") "
Oct 29 21:55:41.426221 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.426215    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istiod-ca-cert\") pod \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\" (UID: \"ac4fac5d-8644-4da7-82a0-aea07ee3f699\") "
Oct 29 21:55:41.429007 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.428832    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-credential-socket" pod "ac4fac5d-8644-4da7-82a0-aea07ee3f699" (UID: "ac4fac5d-8644-4da7-82a0-aea07ee3f699"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:41.429007 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.428923    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-data" pod "ac4fac5d-8644-4da7-82a0-aea07ee3f699" (UID: "ac4fac5d-8644-4da7-82a0-aea07ee3f699"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:41.429349 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.429320    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-workload-certs" pod "ac4fac5d-8644-4da7-82a0-aea07ee3f699" (UID: "ac4fac5d-8644-4da7-82a0-aea07ee3f699"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:41.429689 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.429669    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-workload-socket" pod "ac4fac5d-8644-4da7-82a0-aea07ee3f699" (UID: "ac4fac5d-8644-4da7-82a0-aea07ee3f699"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:41.429943 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.429924    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istiod-ca-cert" pod "ac4fac5d-8644-4da7-82a0-aea07ee3f699" (UID: "ac4fac5d-8644-4da7-82a0-aea07ee3f699"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:41.432103 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.432078    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-envoy" pod "ac4fac5d-8644-4da7-82a0-aea07ee3f699" (UID: "ac4fac5d-8644-4da7-82a0-aea07ee3f699"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:41.432407 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.432387    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-token" pod "ac4fac5d-8644-4da7-82a0-aea07ee3f699" (UID: "ac4fac5d-8644-4da7-82a0-aea07ee3f699"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:41.432506 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.432463    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac4fac5d-8644-4da7-82a0-aea07ee3f699-kube-api-access-xfx9w" pod "ac4fac5d-8644-4da7-82a0-aea07ee3f699" (UID: "ac4fac5d-8644-4da7-82a0-aea07ee3f699"). InnerVolumeSpecName "kube-api-access-xfx9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:41.432566 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.432543    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-podinfo" pod "ac4fac5d-8644-4da7-82a0-aea07ee3f699" (UID: "ac4fac5d-8644-4da7-82a0-aea07ee3f699"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 21:55:41.527802 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.527725    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:41.527802 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.527753    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:41.527802 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.527765    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:41.527802 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.527775    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:41.527802 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.527783    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:41.527802 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.527792    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:41.527802 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.527801    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:41.527802 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.527809    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac4fac5d-8644-4da7-82a0-aea07ee3f699-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:41.528172 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.527817    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfx9w\" (UniqueName: \"kubernetes.io/projected/ac4fac5d-8644-4da7-82a0-aea07ee3f699-kube-api-access-xfx9w\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:41.999735 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.999704    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr"
Oct 29 21:55:41.999735 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.999727    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr" event={"ID":"d91e7562-34b9-4350-896a-c95260f5c07a","Type":"ContainerDied","Data":"403c616c85beb688aae992f306e51d3e4ce6438ba81675d2f562ad54525a596d"}
Oct 29 21:55:42.000310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:41.999798    4962 scope.go:122] "RemoveContainer" containerID="ae2f1fbec7120f85dfe25017f780112ab7486fab839439328db484933c58bceb"
Oct 29 21:55:42.001595 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.001572    4962 generic.go:356] "Generic (PLEG): container finished" podID="ac4fac5d-8644-4da7-82a0-aea07ee3f699" containerID="c9475af40eca959c9e42ac9ede5f83879427c452c4486b1ddbaf29b012046677" exitCode=0
Oct 29 21:55:42.001698 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.001634    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"
Oct 29 21:55:42.001770 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.001751    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc" event={"ID":"ac4fac5d-8644-4da7-82a0-aea07ee3f699","Type":"ContainerDied","Data":"c9475af40eca959c9e42ac9ede5f83879427c452c4486b1ddbaf29b012046677"}
Oct 29 21:55:42.001878 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.001798    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc" event={"ID":"ac4fac5d-8644-4da7-82a0-aea07ee3f699","Type":"ContainerDied","Data":"42957d35c93166c022dab2e6f7d120026a9f078370777e345982183bbc2f5aba"}
Oct 29 21:55:42.011081 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.011066    4962 scope.go:122] "RemoveContainer" containerID="c9475af40eca959c9e42ac9ede5f83879427c452c4486b1ddbaf29b012046677"
Oct 29 21:55:42.019379 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.019361    4962 scope.go:122] "RemoveContainer" containerID="1c1e0a166dd6749274c22ddd84d5179566f0977cbe88af58d7de628ce5d6b763"
Oct 29 21:55:42.027463 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.027449    4962 scope.go:122] "RemoveContainer" containerID="c9475af40eca959c9e42ac9ede5f83879427c452c4486b1ddbaf29b012046677"
Oct 29 21:55:42.027723 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:55:42.027703    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9475af40eca959c9e42ac9ede5f83879427c452c4486b1ddbaf29b012046677\": container with ID starting with c9475af40eca959c9e42ac9ede5f83879427c452c4486b1ddbaf29b012046677 not found: ID does not exist" containerID="c9475af40eca959c9e42ac9ede5f83879427c452c4486b1ddbaf29b012046677"
Oct 29 21:55:42.027767 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.027731    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9475af40eca959c9e42ac9ede5f83879427c452c4486b1ddbaf29b012046677"} err="failed to get container status \"c9475af40eca959c9e42ac9ede5f83879427c452c4486b1ddbaf29b012046677\": rpc error: code = NotFound desc = could not find container \"c9475af40eca959c9e42ac9ede5f83879427c452c4486b1ddbaf29b012046677\": container with ID starting with c9475af40eca959c9e42ac9ede5f83879427c452c4486b1ddbaf29b012046677 not found: ID does not exist"
Oct 29 21:55:42.027767 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.027747    4962 scope.go:122] "RemoveContainer" containerID="1c1e0a166dd6749274c22ddd84d5179566f0977cbe88af58d7de628ce5d6b763"
Oct 29 21:55:42.028009 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:55:42.027987    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c1e0a166dd6749274c22ddd84d5179566f0977cbe88af58d7de628ce5d6b763\": container with ID starting with 1c1e0a166dd6749274c22ddd84d5179566f0977cbe88af58d7de628ce5d6b763 not found: ID does not exist" containerID="1c1e0a166dd6749274c22ddd84d5179566f0977cbe88af58d7de628ce5d6b763"
Oct 29 21:55:42.028093 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.028013    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c1e0a166dd6749274c22ddd84d5179566f0977cbe88af58d7de628ce5d6b763"} err="failed to get container status \"1c1e0a166dd6749274c22ddd84d5179566f0977cbe88af58d7de628ce5d6b763\": rpc error: code = NotFound desc = could not find container \"1c1e0a166dd6749274c22ddd84d5179566f0977cbe88af58d7de628ce5d6b763\": container with ID starting with 1c1e0a166dd6749274c22ddd84d5179566f0977cbe88af58d7de628ce5d6b763 not found: ID does not exist"
Oct 29 21:55:42.053608 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.053587    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"]
Oct 29 21:55:42.058461 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.058438    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-64505fc5-openshift-default-75fcf889df-f5jfc"]
Oct 29 21:55:42.106157 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.106133    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr"]
Oct 29 21:55:42.108639 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.108613    4962 patch_prober.go:29] interesting pod/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 29 21:55:42.108772 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.108672    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh" podUID="10a9a83c-2b74-4cba-8011-63a7f4f2f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 29 21:55:42.117822 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.117802    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f5bae148-f84bc59f5-pbghr"]
Oct 29 21:55:42.936502 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.936471    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"]
Oct 29 21:55:42.936763 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.936697    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm" podUID="590636b8-d3a4-4c12-9c93-7755dbdf4f37" containerName="istio-proxy" containerID="cri-o://1a5b3808bb616bccbecdbaf20f8ea9049d7e921f19f465033c6ef3cf25f3cae7" gracePeriod=30
Oct 29 21:55:42.948994 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.948967    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq"]
Oct 29 21:55:42.949418 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.949404    4962 cpu_manager.go:414] "containerMap: removing container" podUID="ac4fac5d-8644-4da7-82a0-aea07ee3f699" containerName="istio-proxy"
Oct 29 21:55:42.949474 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.949420    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac4fac5d-8644-4da7-82a0-aea07ee3f699" containerName="istio-proxy"
Oct 29 21:55:42.949474 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.949434    4962 cpu_manager.go:414] "containerMap: removing container" podUID="eb1dcd64-e3c0-4ac0-8941-c0845dba1979" containerName="payload-pre-processing"
Oct 29 21:55:42.949474 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.949439    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb1dcd64-e3c0-4ac0-8941-c0845dba1979" containerName="payload-pre-processing"
Oct 29 21:55:42.949474 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.949455    4962 cpu_manager.go:414] "containerMap: removing container" podUID="ac4fac5d-8644-4da7-82a0-aea07ee3f699" containerName="istio-proxy"
Oct 29 21:55:42.949474 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.949460    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac4fac5d-8644-4da7-82a0-aea07ee3f699" containerName="istio-proxy"
Oct 29 21:55:42.949474 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.949466    4962 cpu_manager.go:414] "containerMap: removing container" podUID="d91e7562-34b9-4350-896a-c95260f5c07a" containerName="maas-api"
Oct 29 21:55:42.949474 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.949471    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d91e7562-34b9-4350-896a-c95260f5c07a" containerName="maas-api"
Oct 29 21:55:42.949675 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.949492    4962 cpu_manager.go:414] "containerMap: removing container" podUID="ad9d9073-c42c-4c70-90a5-5971211c41b2" containerName="payload-processing"
Oct 29 21:55:42.949675 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.949496    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad9d9073-c42c-4c70-90a5-5971211c41b2" containerName="payload-processing"
Oct 29 21:55:42.949675 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.949503    4962 cpu_manager.go:414] "containerMap: removing container" podUID="0a47e418-7c3c-4f35-a423-ac4296c9c8a7" containerName="payload-processing"
Oct 29 21:55:42.949675 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.949507    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a47e418-7c3c-4f35-a423-ac4296c9c8a7" containerName="payload-processing"
Oct 29 21:55:42.949675 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.949514    4962 cpu_manager.go:414] "containerMap: removing container" podUID="d62760e8-caa9-49a1-976d-c56196ecd567" containerName="maas-api"
Oct 29 21:55:42.949675 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.949518    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d62760e8-caa9-49a1-976d-c56196ecd567" containerName="maas-api"
Oct 29 21:55:42.949675 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.949575    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac4fac5d-8644-4da7-82a0-aea07ee3f699" containerName="istio-proxy"
Oct 29 21:55:42.949675 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.949582    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb1dcd64-e3c0-4ac0-8941-c0845dba1979" containerName="payload-pre-processing"
Oct 29 21:55:42.949675 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.949591    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="d91e7562-34b9-4350-896a-c95260f5c07a" containerName="maas-api"
Oct 29 21:55:42.949675 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.949600    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad9d9073-c42c-4c70-90a5-5971211c41b2" containerName="payload-processing"
Oct 29 21:55:42.949675 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.949605    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a47e418-7c3c-4f35-a423-ac4296c9c8a7" containerName="payload-processing"
Oct 29 21:55:42.949675 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.949610    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="d62760e8-caa9-49a1-976d-c56196ecd567" containerName="maas-api"
Oct 29 21:55:42.953996 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.953979    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq"
Oct 29 21:55:42.968825 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.968808    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-a940bc\""
Oct 29 21:55:42.968932 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:42.968915    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-a940bc-dockercfg-jw48c\""
Oct 29 21:55:43.002337 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.000797    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq"]
Oct 29 21:55:43.042104 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.042066    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae18ff3-cbb6-4399-92e5-80145ffac570-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq\" (UID: \"fae18ff3-cbb6-4399-92e5-80145ffac570\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq"
Oct 29 21:55:43.042543 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.042517    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68bm9\" (UniqueName: \"kubernetes.io/projected/fae18ff3-cbb6-4399-92e5-80145ffac570-kube-api-access-68bm9\") pod \"payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq\" (UID: \"fae18ff3-cbb6-4399-92e5-80145ffac570\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq"
Oct 29 21:55:43.064592 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.064534    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr"]
Oct 29 21:55:43.065439 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.065415    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac4fac5d-8644-4da7-82a0-aea07ee3f699" containerName="istio-proxy"
Oct 29 21:55:43.096393 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.091362    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr"
Oct 29 21:55:43.128342 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.121487    4962 patch_prober.go:29] interesting pod/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 29 21:55:43.128342 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.121540    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh" podUID="10a9a83c-2b74-4cba-8011-63a7f4f2f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 29 21:55:43.137747 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.136761    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac4fac5d-8644-4da7-82a0-aea07ee3f699" path="/var/lib/kubelet/pods/ac4fac5d-8644-4da7-82a0-aea07ee3f699/volumes"
Oct 29 21:55:43.137747 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.137405    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d91e7562-34b9-4350-896a-c95260f5c07a" path="/var/lib/kubelet/pods/d91e7562-34b9-4350-896a-c95260f5c07a/volumes"
Oct 29 21:55:43.137976 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.137784    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr"]
Oct 29 21:55:43.143363 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.143180    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68bm9\" (UniqueName: \"kubernetes.io/projected/fae18ff3-cbb6-4399-92e5-80145ffac570-kube-api-access-68bm9\") pod \"payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq\" (UID: \"fae18ff3-cbb6-4399-92e5-80145ffac570\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq"
Oct 29 21:55:43.143363 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.143269    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae18ff3-cbb6-4399-92e5-80145ffac570-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq\" (UID: \"fae18ff3-cbb6-4399-92e5-80145ffac570\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq"
Oct 29 21:55:43.144237 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.144038    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae18ff3-cbb6-4399-92e5-80145ffac570-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq\" (UID: \"fae18ff3-cbb6-4399-92e5-80145ffac570\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq"
Oct 29 21:55:43.168516 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.168486    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68bm9\" (UniqueName: \"kubernetes.io/projected/fae18ff3-cbb6-4399-92e5-80145ffac570-kube-api-access-68bm9\") pod \"payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq\" (UID: \"fae18ff3-cbb6-4399-92e5-80145ffac570\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq"
Oct 29 21:55:43.244978 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.244773    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bmc6\" (UniqueName: \"kubernetes.io/projected/d0b72137-f812-4900-adce-9b88820a6473-kube-api-access-2bmc6\") pod \"payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr\" (UID: \"d0b72137-f812-4900-adce-9b88820a6473\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr"
Oct 29 21:55:43.244978 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.244924    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0b72137-f812-4900-adce-9b88820a6473-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr\" (UID: \"d0b72137-f812-4900-adce-9b88820a6473\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr"
Oct 29 21:55:43.267457 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.267435    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq"
Oct 29 21:55:43.346401 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.345976    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bmc6\" (UniqueName: \"kubernetes.io/projected/d0b72137-f812-4900-adce-9b88820a6473-kube-api-access-2bmc6\") pod \"payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr\" (UID: \"d0b72137-f812-4900-adce-9b88820a6473\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr"
Oct 29 21:55:43.346401 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.346110    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0b72137-f812-4900-adce-9b88820a6473-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr\" (UID: \"d0b72137-f812-4900-adce-9b88820a6473\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr"
Oct 29 21:55:43.346895 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.346872    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0b72137-f812-4900-adce-9b88820a6473-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr\" (UID: \"d0b72137-f812-4900-adce-9b88820a6473\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr"
Oct 29 21:55:43.397345 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.397272    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bmc6\" (UniqueName: \"kubernetes.io/projected/d0b72137-f812-4900-adce-9b88820a6473-kube-api-access-2bmc6\") pod \"payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr\" (UID: \"d0b72137-f812-4900-adce-9b88820a6473\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr"
Oct 29 21:55:43.459815 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.459788    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr"
Oct 29 21:55:43.603330 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.601137    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq"]
Oct 29 21:55:43.608836 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:55:43.608806    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae18ff3_cbb6_4399_92e5_80145ffac570.slice/crio-eb3d96bc8743fbbeac4076571be8d8d5dd511ded3bd278e67d062fde45677785 WatchSource:0}: Error finding container eb3d96bc8743fbbeac4076571be8d8d5dd511ded3bd278e67d062fde45677785: Status 404 returned error can't find the container with id eb3d96bc8743fbbeac4076571be8d8d5dd511ded3bd278e67d062fde45677785
Oct 29 21:55:43.826982 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.826894    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f97668b9-65c448566d-bkh29" podUID="e3c6f723-516c-411d-88c4-798c15811838" containerName="payload-processing" containerID="cri-o://0e313e7eeba2df02eed89859252ba3658f93085f3c1b1a6a00a3c4734cc750d4" gracePeriod=25
Oct 29 21:55:43.830210 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.830185    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr"]
Oct 29 21:55:43.831771 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:55:43.831741    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0b72137_f812_4900_adce_9b88820a6473.slice/crio-6336a14ee4e5b6c9c3a5395501d1941b6dccd0f3ab9625d9eea3845f42adb3ec WatchSource:0}: Error finding container 6336a14ee4e5b6c9c3a5395501d1941b6dccd0f3ab9625d9eea3845f42adb3ec: Status 404 returned error can't find the container with id 6336a14ee4e5b6c9c3a5395501d1941b6dccd0f3ab9625d9eea3845f42adb3ec
Oct 29 21:55:43.865609 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:43.865582    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7" podUID="7c51f1c6-c2da-492c-ae4c-11979c06e914" containerName="payload-pre-processing" containerID="cri-o://200df1cd6835cf4612e1144f51213f126fb6b5676144e17629998d1d566ceb99" gracePeriod=25
Oct 29 21:55:44.018949 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.018918    4962 generic.go:356] "Generic (PLEG): container finished" podID="7c51f1c6-c2da-492c-ae4c-11979c06e914" containerID="200df1cd6835cf4612e1144f51213f126fb6b5676144e17629998d1d566ceb99" exitCode=0
Oct 29 21:55:44.019645 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.018965    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7" event={"ID":"7c51f1c6-c2da-492c-ae4c-11979c06e914","Type":"ContainerDied","Data":"200df1cd6835cf4612e1144f51213f126fb6b5676144e17629998d1d566ceb99"}
Oct 29 21:55:44.020972 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.020950    4962 generic.go:356] "Generic (PLEG): container finished" podID="e3c6f723-516c-411d-88c4-798c15811838" containerID="0e313e7eeba2df02eed89859252ba3658f93085f3c1b1a6a00a3c4734cc750d4" exitCode=0
Oct 29 21:55:44.021091 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.021036    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f97668b9-65c448566d-bkh29" event={"ID":"e3c6f723-516c-411d-88c4-798c15811838","Type":"ContainerDied","Data":"0e313e7eeba2df02eed89859252ba3658f93085f3c1b1a6a00a3c4734cc750d4"}
Oct 29 21:55:44.023023 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.022997    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq" event={"ID":"fae18ff3-cbb6-4399-92e5-80145ffac570","Type":"ContainerStarted","Data":"bb368e9a8a6eeac04117dfe2a04806d067eb32548f0de23252260ea6539afda3"}
Oct 29 21:55:44.023157 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.023139    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq" event={"ID":"fae18ff3-cbb6-4399-92e5-80145ffac570","Type":"ContainerStarted","Data":"eb3d96bc8743fbbeac4076571be8d8d5dd511ded3bd278e67d062fde45677785"}
Oct 29 21:55:44.023314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.023282    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq"
Oct 29 21:55:44.025051 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.025024    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr" event={"ID":"d0b72137-f812-4900-adce-9b88820a6473","Type":"ContainerStarted","Data":"6336a14ee4e5b6c9c3a5395501d1941b6dccd0f3ab9625d9eea3845f42adb3ec"}
Oct 29 21:55:44.025370 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.025326    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr"
Oct 29 21:55:44.108992 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.108615    4962 patch_prober.go:29] interesting pod/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 29 21:55:44.108992 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.108665    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh" podUID="10a9a83c-2b74-4cba-8011-63a7f4f2f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 29 21:55:44.127466 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.127408    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr" podStartSLOduration=1.127389886 podStartE2EDuration="1.127389886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:44.1196185 +0000 UTC m=+565.192575464" watchObservedRunningTime="2026-10-29 21:55:44.127389886 +0000 UTC m=+565.200347044"
Oct 29 21:55:44.128620 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.128579    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq" podStartSLOduration=2.128564618 podStartE2EDuration="2.128564618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:44.06892866 +0000 UTC m=+565.141885653" watchObservedRunningTime="2026-10-29 21:55:44.128564618 +0000 UTC m=+565.201521581"
Oct 29 21:55:44.170106 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.170079    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f97668b9-65c448566d-bkh29"
Oct 29 21:55:44.247043 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.247008    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"]
Oct 29 21:55:44.247780 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.247756    4962 cpu_manager.go:414] "containerMap: removing container" podUID="e3c6f723-516c-411d-88c4-798c15811838" containerName="payload-processing"
Oct 29 21:55:44.247780 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.247778    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c6f723-516c-411d-88c4-798c15811838" containerName="payload-processing"
Oct 29 21:55:44.247936 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.247919    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c6f723-516c-411d-88c4-798c15811838" containerName="payload-processing"
Oct 29 21:55:44.254590 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.254566    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.259559 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.259538    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c6f723-516c-411d-88c4-798c15811838-kube-api-access-kgzdb\" (UniqueName: \"kubernetes.io/projected/e3c6f723-516c-411d-88c4-798c15811838-kube-api-access-kgzdb\") pod \"e3c6f723-516c-411d-88c4-798c15811838\" (UID: \"e3c6f723-516c-411d-88c4-798c15811838\") "
Oct 29 21:55:44.259716 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.259701    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3c6f723-516c-411d-88c4-798c15811838-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c6f723-516c-411d-88c4-798c15811838-plugins-config-volume\") pod \"e3c6f723-516c-411d-88c4-798c15811838\" (UID: \"e3c6f723-516c-411d-88c4-798c15811838\") "
Oct 29 21:55:44.260108 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.260083    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c6f723-516c-411d-88c4-798c15811838-plugins-config-volume" pod "e3c6f723-516c-411d-88c4-798c15811838" (UID: "e3c6f723-516c-411d-88c4-798c15811838"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:44.261692 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.261666    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c6f723-516c-411d-88c4-798c15811838-kube-api-access-kgzdb" pod "e3c6f723-516c-411d-88c4-798c15811838" (UID: "e3c6f723-516c-411d-88c4-798c15811838"). InnerVolumeSpecName "kube-api-access-kgzdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:44.303352 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.303326    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b31f3f-openshift-default-dockercfg-ddh7z\""
Oct 29 21:55:44.305072 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.303554    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"]
Oct 29 21:55:44.321873 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.321850    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7"
Oct 29 21:55:44.360492 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.360437    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.360492 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.360473    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.360668 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.360572    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.360668 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.360616    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.360773 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.360723    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.360814 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.360783    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.360852 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.360828    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltzl8\" (UniqueName: \"kubernetes.io/projected/75cf3e4f-0488-4210-ab72-17b7afdeda6a-kube-api-access-ltzl8\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.360899 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.360861    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.360899 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.360879    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.361056 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.361041    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3c6f723-516c-411d-88c4-798c15811838-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:44.361104 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.361063    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgzdb\" (UniqueName: \"kubernetes.io/projected/e3c6f723-516c-411d-88c4-798c15811838-kube-api-access-kgzdb\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:44.461635 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.461597    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c51f1c6-c2da-492c-ae4c-11979c06e914-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c51f1c6-c2da-492c-ae4c-11979c06e914-plugins-config-volume\") pod \"7c51f1c6-c2da-492c-ae4c-11979c06e914\" (UID: \"7c51f1c6-c2da-492c-ae4c-11979c06e914\") "
Oct 29 21:55:44.461804 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.461696    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c51f1c6-c2da-492c-ae4c-11979c06e914-kube-api-access-rfbgf\" (UniqueName: \"kubernetes.io/projected/7c51f1c6-c2da-492c-ae4c-11979c06e914-kube-api-access-rfbgf\") pod \"7c51f1c6-c2da-492c-ae4c-11979c06e914\" (UID: \"7c51f1c6-c2da-492c-ae4c-11979c06e914\") "
Oct 29 21:55:44.461939 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.461915    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c51f1c6-c2da-492c-ae4c-11979c06e914-plugins-config-volume" pod "7c51f1c6-c2da-492c-ae4c-11979c06e914" (UID: "7c51f1c6-c2da-492c-ae4c-11979c06e914"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:44.461986 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.461925    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.461986 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.461979    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.462057 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.462033    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.462114 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.462059    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.462172 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.462112    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltzl8\" (UniqueName: \"kubernetes.io/projected/75cf3e4f-0488-4210-ab72-17b7afdeda6a-kube-api-access-ltzl8\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.462172 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.462161    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.462253 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.462206    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.462320 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.462251    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.462320 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.462278    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.462411 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.462331    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.462411 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.462402    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c51f1c6-c2da-492c-ae4c-11979c06e914-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:44.462509 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.462485    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.462629 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.462564    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.462784 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.462760    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.463131 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.463110    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.464469 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.464443    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c51f1c6-c2da-492c-ae4c-11979c06e914-kube-api-access-rfbgf" pod "7c51f1c6-c2da-492c-ae4c-11979c06e914" (UID: "7c51f1c6-c2da-492c-ae4c-11979c06e914"). InnerVolumeSpecName "kube-api-access-rfbgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:44.464925 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.464872    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.465025 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.464980    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.494773 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.494706    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.496028 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.495617    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltzl8\" (UniqueName: \"kubernetes.io/projected/75cf3e4f-0488-4210-ab72-17b7afdeda6a-kube-api-access-ltzl8\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.563807 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.563757    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfbgf\" (UniqueName: \"kubernetes.io/projected/7c51f1c6-c2da-492c-ae4c-11979c06e914-kube-api-access-rfbgf\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:44.620267 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.620179    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:44.955945 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.955915    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"]
Oct 29 21:55:44.957661 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:55:44.957635    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75cf3e4f_0488_4210_ab72_17b7afdeda6a.slice/crio-27b698f1abec76abc65b36051054fc42f0e41c7e2700a6681bb798bb0fa5f036 WatchSource:0}: Error finding container 27b698f1abec76abc65b36051054fc42f0e41c7e2700a6681bb798bb0fa5f036: Status 404 returned error can't find the container with id 27b698f1abec76abc65b36051054fc42f0e41c7e2700a6681bb798bb0fa5f036
Oct 29 21:55:44.959940 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.959914    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:55:44.959995 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.959972    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:55:44.960036 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:44.960001    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:55:45.034150 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:45.034119    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr" event={"ID":"d0b72137-f812-4900-adce-9b88820a6473","Type":"ContainerStarted","Data":"367cb5b6741d771dcc9eaeb7df1b875a741eed9f41c907dc2489aaf283ecbb18"}
Oct 29 21:55:45.035513 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:45.035491    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7" event={"ID":"7c51f1c6-c2da-492c-ae4c-11979c06e914","Type":"ContainerDied","Data":"aeed3c4c6b7c14c9f2c044614259fb2428e60fb0069980e586890c5714932174"}
Oct 29 21:55:45.035513 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:45.035506    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7"
Oct 29 21:55:45.035681 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:45.035547    4962 scope.go:122] "RemoveContainer" containerID="200df1cd6835cf4612e1144f51213f126fb6b5676144e17629998d1d566ceb99"
Oct 29 21:55:45.036932 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:45.036895    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f97668b9-65c448566d-bkh29"
Oct 29 21:55:45.036932 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:45.036916    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f97668b9-65c448566d-bkh29" event={"ID":"e3c6f723-516c-411d-88c4-798c15811838","Type":"ContainerDied","Data":"35ad2c7fc6379bfed631c09c7d05572b96740dfa7a6ceba48bf046db34a35be2"}
Oct 29 21:55:45.040877 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:45.040855    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf" event={"ID":"75cf3e4f-0488-4210-ab72-17b7afdeda6a","Type":"ContainerStarted","Data":"27b698f1abec76abc65b36051054fc42f0e41c7e2700a6681bb798bb0fa5f036"}
Oct 29 21:55:45.045680 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:45.045650    4962 scope.go:122] "RemoveContainer" containerID="0e313e7eeba2df02eed89859252ba3658f93085f3c1b1a6a00a3c4734cc750d4"
Oct 29 21:55:45.115346 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:45.109713    4962 patch_prober.go:29] interesting pod/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 29 21:55:45.115346 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:45.109779    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh" podUID="10a9a83c-2b74-4cba-8011-63a7f4f2f134" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 29 21:55:45.142120 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:45.142077    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7"]
Oct 29 21:55:45.158364 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:45.158333    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f97668b9-c7998ccf5-bmsb7"]
Oct 29 21:55:45.196591 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:45.196561    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f97668b9-65c448566d-bkh29"]
Oct 29 21:55:45.204932 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:45.204855    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f97668b9-65c448566d-bkh29"]
Oct 29 21:55:46.049926 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:46.049892    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf" event={"ID":"75cf3e4f-0488-4210-ab72-17b7afdeda6a","Type":"ContainerStarted","Data":"09a8d0a86523179a06b9f2864e8a2ed76f2894553b2d17cad788c3c674d66ab6"}
Oct 29 21:55:46.108824 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:46.108767    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf" podStartSLOduration=2.108752849 podStartE2EDuration="2.108752849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:46.108380042 +0000 UTC m=+567.181337025" watchObservedRunningTime="2026-10-29 21:55:46.108752849 +0000 UTC m=+567.181709813"
Oct 29 21:55:46.329409 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:46.329333    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:55:46.329811 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:46.329787    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:55:46.330709 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:46.330685    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:55:46.620678 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:46.620579    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:46.630404 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:46.630345    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:47.057306 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:47.057267    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c51f1c6-c2da-492c-ae4c-11979c06e914" path="/var/lib/kubelet/pods/7c51f1c6-c2da-492c-ae4c-11979c06e914/volumes"
Oct 29 21:55:47.057660 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:47.057650    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3c6f723-516c-411d-88c4-798c15811838" path="/var/lib/kubelet/pods/e3c6f723-516c-411d-88c4-798c15811838/volumes"
Oct 29 21:55:47.057923 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:47.057909    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:47.057972 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:47.057966    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:55:48.063164 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.063138    4962 generic.go:356] "Generic (PLEG): container finished" podID="590636b8-d3a4-4c12-9c93-7755dbdf4f37" containerID="1a5b3808bb616bccbecdbaf20f8ea9049d7e921f19f465033c6ef3cf25f3cae7" exitCode=0
Oct 29 21:55:48.063512 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.063213    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm" event={"ID":"590636b8-d3a4-4c12-9c93-7755dbdf4f37","Type":"ContainerDied","Data":"1a5b3808bb616bccbecdbaf20f8ea9049d7e921f19f465033c6ef3cf25f3cae7"}
Oct 29 21:55:48.063512 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.063298    4962 scope.go:122] "RemoveContainer" containerID="9cecdea73b7afddf02f82b32c4b7a3dceae967f59558bd45f1c9ba2b1ff8dc79"
Oct 29 21:55:48.210409 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.210354    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:55:48.335940 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.335911    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-envoy\") pod \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") "
Oct 29 21:55:48.336079 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.335948    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-workload-certs\") pod \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") "
Oct 29 21:55:48.336079 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.335992    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-credential-socket\") pod \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") "
Oct 29 21:55:48.336079 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.336041    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-podinfo\") pod \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") "
Oct 29 21:55:48.336200 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.336088    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istiod-ca-cert\") pod \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") "
Oct 29 21:55:48.336200 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.336124    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-data\") pod \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") "
Oct 29 21:55:48.336200 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.336181    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/590636b8-d3a4-4c12-9c93-7755dbdf4f37-kube-api-access-zx6jz\" (UniqueName: \"kubernetes.io/projected/590636b8-d3a4-4c12-9c93-7755dbdf4f37-kube-api-access-zx6jz\") pod \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") "
Oct 29 21:55:48.336456 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.336248    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-token\" (UniqueName: \"kubernetes.io/projected/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-token\") pod \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") "
Oct 29 21:55:48.336456 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.336277    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-workload-socket\") pod \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\" (UID: \"590636b8-d3a4-4c12-9c93-7755dbdf4f37\") "
Oct 29 21:55:48.336456 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.336280    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-workload-certs" pod "590636b8-d3a4-4c12-9c93-7755dbdf4f37" (UID: "590636b8-d3a4-4c12-9c93-7755dbdf4f37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:48.336456 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.336360    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-credential-socket" pod "590636b8-d3a4-4c12-9c93-7755dbdf4f37" (UID: "590636b8-d3a4-4c12-9c93-7755dbdf4f37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:48.336660 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.336626    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-workload-socket" pod "590636b8-d3a4-4c12-9c93-7755dbdf4f37" (UID: "590636b8-d3a4-4c12-9c93-7755dbdf4f37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:48.336717 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.336648    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istiod-ca-cert" pod "590636b8-d3a4-4c12-9c93-7755dbdf4f37" (UID: "590636b8-d3a4-4c12-9c93-7755dbdf4f37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:48.336863 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.336847    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:48.336929 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.336869    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:48.336929 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.336883    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:48.336929 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.336897    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:48.337058 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.337039    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-data" pod "590636b8-d3a4-4c12-9c93-7755dbdf4f37" (UID: "590636b8-d3a4-4c12-9c93-7755dbdf4f37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:48.338390 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.338366    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-envoy" pod "590636b8-d3a4-4c12-9c93-7755dbdf4f37" (UID: "590636b8-d3a4-4c12-9c93-7755dbdf4f37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:48.338765 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.338748    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-token" pod "590636b8-d3a4-4c12-9c93-7755dbdf4f37" (UID: "590636b8-d3a4-4c12-9c93-7755dbdf4f37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:48.338879 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.338855    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-podinfo" pod "590636b8-d3a4-4c12-9c93-7755dbdf4f37" (UID: "590636b8-d3a4-4c12-9c93-7755dbdf4f37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 21:55:48.339360 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.339338    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590636b8-d3a4-4c12-9c93-7755dbdf4f37-kube-api-access-zx6jz" pod "590636b8-d3a4-4c12-9c93-7755dbdf4f37" (UID: "590636b8-d3a4-4c12-9c93-7755dbdf4f37"). InnerVolumeSpecName "kube-api-access-zx6jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:48.355903 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.355880    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"]
Oct 29 21:55:48.356445 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.356429    4962 cpu_manager.go:414] "containerMap: removing container" podUID="590636b8-d3a4-4c12-9c93-7755dbdf4f37" containerName="istio-proxy"
Oct 29 21:55:48.356556 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.356447    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="590636b8-d3a4-4c12-9c93-7755dbdf4f37" containerName="istio-proxy"
Oct 29 21:55:48.356556 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.356465    4962 cpu_manager.go:414] "containerMap: removing container" podUID="590636b8-d3a4-4c12-9c93-7755dbdf4f37" containerName="istio-proxy"
Oct 29 21:55:48.356556 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.356473    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="590636b8-d3a4-4c12-9c93-7755dbdf4f37" containerName="istio-proxy"
Oct 29 21:55:48.356556 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.356487    4962 cpu_manager.go:414] "containerMap: removing container" podUID="7c51f1c6-c2da-492c-ae4c-11979c06e914" containerName="payload-pre-processing"
Oct 29 21:55:48.356556 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.356495    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c51f1c6-c2da-492c-ae4c-11979c06e914" containerName="payload-pre-processing"
Oct 29 21:55:48.356794 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.356609    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="590636b8-d3a4-4c12-9c93-7755dbdf4f37" containerName="istio-proxy"
Oct 29 21:55:48.356794 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.356622    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="590636b8-d3a4-4c12-9c93-7755dbdf4f37" containerName="istio-proxy"
Oct 29 21:55:48.356794 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.356634    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c51f1c6-c2da-492c-ae4c-11979c06e914" containerName="payload-pre-processing"
Oct 29 21:55:48.359976 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.359959    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.374485 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.374458    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-a9def7-openshift-default-dockercfg-qflmj\""
Oct 29 21:55:48.385409 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.385382    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"]
Oct 29 21:55:48.437840 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.437818    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.437933 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.437862    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws6dp\" (UniqueName: \"kubernetes.io/projected/d71de0ec-1eaa-481f-8cee-cb256e330136-kube-api-access-ws6dp\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.437933 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.437924    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.438013 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.437944    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.438013 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.437966    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d71de0ec-1eaa-481f-8cee-cb256e330136-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.438076 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.438020    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.438109 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.438082    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.438145 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.438106    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.438203 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.438192    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.438258 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.438249    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:48.438300 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.438261    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:48.438331 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.438309    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:48.438365 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.438329    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx6jz\" (UniqueName: \"kubernetes.io/projected/590636b8-d3a4-4c12-9c93-7755dbdf4f37-kube-api-access-zx6jz\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:48.438365 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.438341    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/590636b8-d3a4-4c12-9c93-7755dbdf4f37-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:48.541890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.541787    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws6dp\" (UniqueName: \"kubernetes.io/projected/d71de0ec-1eaa-481f-8cee-cb256e330136-kube-api-access-ws6dp\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.549324 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.542261    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.549324 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.542340    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.549324 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.542380    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.549324 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.542415    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d71de0ec-1eaa-481f-8cee-cb256e330136-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.549324 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.542452    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.549324 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.542505    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.549324 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.542541    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.549324 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.542632    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.549324 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.542703    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.549324 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.542747    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.549324 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.543130    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.549324 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.543867    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d71de0ec-1eaa-481f-8cee-cb256e330136-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.549324 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.544119    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.549324 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.545940    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.549324 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.546765    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.567267 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.567245    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.567543 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.567522    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws6dp\" (UniqueName: \"kubernetes.io/projected/d71de0ec-1eaa-481f-8cee-cb256e330136-kube-api-access-ws6dp\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:48.673036 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:48.673011    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:49.015636 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.015586    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"]
Oct 29 21:55:49.018444 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.018403    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:55:49.018570 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.018515    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:55:49.018636 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.018565    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:55:49.069497 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.069477    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"
Oct 29 21:55:49.069905 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.069510    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm" event={"ID":"590636b8-d3a4-4c12-9c93-7755dbdf4f37","Type":"ContainerDied","Data":"6b7ba12b50139be7566e5cd194da167156673d3d5f083bbffb10b3eadce19d87"}
Oct 29 21:55:49.069905 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.069558    4962 scope.go:122] "RemoveContainer" containerID="1a5b3808bb616bccbecdbaf20f8ea9049d7e921f19f465033c6ef3cf25f3cae7"
Oct 29 21:55:49.071016 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.070993    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv" event={"ID":"d71de0ec-1eaa-481f-8cee-cb256e330136","Type":"ContainerStarted","Data":"1dfb81cf5e3c8a2f5616e698d0d525feed533a64d9ffe0c4fa75273f732e2474"}
Oct 29 21:55:49.123952 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.123916    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"]
Oct 29 21:55:49.134193 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.134166    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f5bae148-openshift-default-79987cf5cc-kcjzm"]
Oct 29 21:55:49.366432 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.366356    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl"]
Oct 29 21:55:49.370233 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.370209    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl"
Oct 29 21:55:49.398496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.398473    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-a940bc-serving-cert\""
Oct 29 21:55:49.405964 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.405936    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl"]
Oct 29 21:55:49.470788 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.470756    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plq5r\" (UniqueName: \"kubernetes.io/projected/49d31a1c-a9bf-413b-baf8-0ce2a310de72-kube-api-access-plq5r\") pod \"maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl\" (UID: \"49d31a1c-a9bf-413b-baf8-0ce2a310de72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl"
Oct 29 21:55:49.470927 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.470807    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49d31a1c-a9bf-413b-baf8-0ce2a310de72-metrics-certs\") pod \"maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl\" (UID: \"49d31a1c-a9bf-413b-baf8-0ce2a310de72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl"
Oct 29 21:55:49.471046 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.471026    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d31a1c-a9bf-413b-baf8-0ce2a310de72-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl\" (UID: \"49d31a1c-a9bf-413b-baf8-0ce2a310de72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl"
Oct 29 21:55:49.471092 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.471080    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49d31a1c-a9bf-413b-baf8-0ce2a310de72-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl\" (UID: \"49d31a1c-a9bf-413b-baf8-0ce2a310de72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl"
Oct 29 21:55:49.546212 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.546176    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"]
Oct 29 21:55:49.550862 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.550841    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.569162 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.569139    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-743a5e-openshift-default-dockercfg-rnvpp\""
Oct 29 21:55:49.572619 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.572597    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plq5r\" (UniqueName: \"kubernetes.io/projected/49d31a1c-a9bf-413b-baf8-0ce2a310de72-kube-api-access-plq5r\") pod \"maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl\" (UID: \"49d31a1c-a9bf-413b-baf8-0ce2a310de72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl"
Oct 29 21:55:49.572727 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.572647    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49d31a1c-a9bf-413b-baf8-0ce2a310de72-metrics-certs\") pod \"maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl\" (UID: \"49d31a1c-a9bf-413b-baf8-0ce2a310de72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl"
Oct 29 21:55:49.572788 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.572767    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d31a1c-a9bf-413b-baf8-0ce2a310de72-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl\" (UID: \"49d31a1c-a9bf-413b-baf8-0ce2a310de72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl"
Oct 29 21:55:49.572844 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.572834    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49d31a1c-a9bf-413b-baf8-0ce2a310de72-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl\" (UID: \"49d31a1c-a9bf-413b-baf8-0ce2a310de72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl"
Oct 29 21:55:49.573382 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.573360    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d31a1c-a9bf-413b-baf8-0ce2a310de72-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl\" (UID: \"49d31a1c-a9bf-413b-baf8-0ce2a310de72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl"
Oct 29 21:55:49.575199 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.575175    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49d31a1c-a9bf-413b-baf8-0ce2a310de72-metrics-certs\") pod \"maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl\" (UID: \"49d31a1c-a9bf-413b-baf8-0ce2a310de72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl"
Oct 29 21:55:49.575282 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.575247    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49d31a1c-a9bf-413b-baf8-0ce2a310de72-maas-api-tls\") pod \"maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl\" (UID: \"49d31a1c-a9bf-413b-baf8-0ce2a310de72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl"
Oct 29 21:55:49.579237 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.579218    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"]
Oct 29 21:55:49.600175 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.600140    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plq5r\" (UniqueName: \"kubernetes.io/projected/49d31a1c-a9bf-413b-baf8-0ce2a310de72-kube-api-access-plq5r\") pod \"maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl\" (UID: \"49d31a1c-a9bf-413b-baf8-0ce2a310de72\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl"
Oct 29 21:55:49.695321 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.674191    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b2ca39-a1c7-45bf-8458-f23215d334b1-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.695321 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.674262    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.695321 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.693283    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl"
Oct 29 21:55:49.696243 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.695723    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.696243 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.695813    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh82l\" (UniqueName: \"kubernetes.io/projected/03b2ca39-a1c7-45bf-8458-f23215d334b1-kube-api-access-lh82l\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.696243 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.695890    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.696243 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.695916    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.696243 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.695965    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.696243 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.696012    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.696243 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.696039    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.797654 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.797627    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.797773 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.797707    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.797773 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.797734    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.797878 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.797788    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b2ca39-a1c7-45bf-8458-f23215d334b1-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.797878 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.797821    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.797988 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.797885    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.797988 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.797945    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh82l\" (UniqueName: \"kubernetes.io/projected/03b2ca39-a1c7-45bf-8458-f23215d334b1-kube-api-access-lh82l\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.798085 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.798010    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.798085 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.798037    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.799151 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.798496    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.799151 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.798864    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b2ca39-a1c7-45bf-8458-f23215d334b1-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.799556 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.799533    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.799750 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.799734    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.799923 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.799909    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.803102 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.803065    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.803566 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.803545    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.822227 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.822204    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.822771 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.822669    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh82l\" (UniqueName: \"kubernetes.io/projected/03b2ca39-a1c7-45bf-8458-f23215d334b1-kube-api-access-lh82l\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:49.864323 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:49.864276    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:50.037799 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:50.037776    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl"]
Oct 29 21:55:50.039673 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:55:50.039643    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d31a1c_a9bf_413b_baf8_0ce2a310de72.slice/crio-c0a861d715160f8e237a645d8f77761469e61f58a2cae3f307c301a82a8027ff WatchSource:0}: Error finding container c0a861d715160f8e237a645d8f77761469e61f58a2cae3f307c301a82a8027ff: Status 404 returned error can't find the container with id c0a861d715160f8e237a645d8f77761469e61f58a2cae3f307c301a82a8027ff
Oct 29 21:55:50.078741 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:50.078707    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv" event={"ID":"d71de0ec-1eaa-481f-8cee-cb256e330136","Type":"ContainerStarted","Data":"7d7c79aa202bea72b52ec17fff7e032f5c0c30df32d9e7ee9b73879951fd5010"}
Oct 29 21:55:50.080087 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:50.080064    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl" event={"ID":"49d31a1c-a9bf-413b-baf8-0ce2a310de72","Type":"ContainerStarted","Data":"c0a861d715160f8e237a645d8f77761469e61f58a2cae3f307c301a82a8027ff"}
Oct 29 21:55:50.126614 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:50.126563    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv" podStartSLOduration=2.126545985 podStartE2EDuration="2.126545985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:50.124254769 +0000 UTC m=+571.197211734" watchObservedRunningTime="2026-10-29 21:55:50.126545985 +0000 UTC m=+571.199502956"
Oct 29 21:55:50.257202 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:50.257181    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"]
Oct 29 21:55:50.259394 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:55:50.259352    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b2ca39_a1c7_45bf_8458_f23215d334b1.slice/crio-0bdddababf5c33e304ffd2a7742fac211ad1d27ed4e96f9a4ab36daeea703f14 WatchSource:0}: Error finding container 0bdddababf5c33e304ffd2a7742fac211ad1d27ed4e96f9a4ab36daeea703f14: Status 404 returned error can't find the container with id 0bdddababf5c33e304ffd2a7742fac211ad1d27ed4e96f9a4ab36daeea703f14
Oct 29 21:55:50.261880 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:50.261845    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:55:50.261993 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:50.261922    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:55:50.261993 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:50.261967    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:55:50.675450 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:50.675377    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:50.683744 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:50.683712    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:51.068265 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.068093    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="590636b8-d3a4-4c12-9c93-7755dbdf4f37" path="/var/lib/kubelet/pods/590636b8-d3a4-4c12-9c93-7755dbdf4f37/volumes"
Oct 29 21:55:51.087916 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.087882    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl" event={"ID":"49d31a1c-a9bf-413b-baf8-0ce2a310de72","Type":"ContainerStarted","Data":"e30b70dbb0efe8d96a328120d9ae91b763e48af18a99975cc3bae263d68e8dce"}
Oct 29 21:55:51.088326 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.087948    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl"
Oct 29 21:55:51.089986 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.089955    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2" event={"ID":"03b2ca39-a1c7-45bf-8458-f23215d334b1","Type":"ContainerStarted","Data":"d54981ad8aa1ebada11c30c0ac5a4d1cf6117480f18fec0001c3f840f139970a"}
Oct 29 21:55:51.090118 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.089998    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2" event={"ID":"03b2ca39-a1c7-45bf-8458-f23215d334b1","Type":"ContainerStarted","Data":"0bdddababf5c33e304ffd2a7742fac211ad1d27ed4e96f9a4ab36daeea703f14"}
Oct 29 21:55:51.090118 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.090110    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:51.091117 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.091098    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:55:51.115975 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.115919    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2"]
Oct 29 21:55:51.116217 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.116128    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2" podUID="0eb99ce9-22ff-4375-9d98-7cdb1a9675d8" containerName="maas-api" containerID="cri-o://c1d9f8375a0ccdc1a1e452f154d496dbdbc2b99ef45398318be05d4aef6beaa8" gracePeriod=30
Oct 29 21:55:51.131519 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.131468    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl" podStartSLOduration=1.80527846 podStartE2EDuration="2.131451613s" totalImagesPullingTime="326.173153ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:51.130243532 +0000 UTC m=+572.203200497" watchObservedRunningTime="2026-10-29 21:55:51.131451613 +0000 UTC m=+572.204408578"
Oct 29 21:55:51.178238 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.178192    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2" podStartSLOduration=2.178179842 podStartE2EDuration="2.178179842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:51.176049143 +0000 UTC m=+572.249006111" watchObservedRunningTime="2026-10-29 21:55:51.178179842 +0000 UTC m=+572.251136864"
Oct 29 21:55:51.463657 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.463631    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2"
Oct 29 21:55:51.470127 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.470104    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-maas-api-tls\") pod \"0eb99ce9-22ff-4375-9d98-7cdb1a9675d8\" (UID: \"0eb99ce9-22ff-4375-9d98-7cdb1a9675d8\") "
Oct 29 21:55:51.470245 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.470198    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-metrics-certs\") pod \"0eb99ce9-22ff-4375-9d98-7cdb1a9675d8\" (UID: \"0eb99ce9-22ff-4375-9d98-7cdb1a9675d8\") "
Oct 29 21:55:51.470348 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.470245    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-kube-api-access-6vn8x\" (UniqueName: \"kubernetes.io/projected/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-kube-api-access-6vn8x\") pod \"0eb99ce9-22ff-4375-9d98-7cdb1a9675d8\" (UID: \"0eb99ce9-22ff-4375-9d98-7cdb1a9675d8\") "
Oct 29 21:55:51.470348 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.470318    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-openshift-service-ca\") pod \"0eb99ce9-22ff-4375-9d98-7cdb1a9675d8\" (UID: \"0eb99ce9-22ff-4375-9d98-7cdb1a9675d8\") "
Oct 29 21:55:51.470705 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.470679    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-openshift-service-ca" pod "0eb99ce9-22ff-4375-9d98-7cdb1a9675d8" (UID: "0eb99ce9-22ff-4375-9d98-7cdb1a9675d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:51.472463 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.472429    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-kube-api-access-6vn8x" pod "0eb99ce9-22ff-4375-9d98-7cdb1a9675d8" (UID: "0eb99ce9-22ff-4375-9d98-7cdb1a9675d8"). InnerVolumeSpecName "kube-api-access-6vn8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:51.472720 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.472699    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-metrics-certs" pod "0eb99ce9-22ff-4375-9d98-7cdb1a9675d8" (UID: "0eb99ce9-22ff-4375-9d98-7cdb1a9675d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:55:51.472779 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.472719    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-maas-api-tls" pod "0eb99ce9-22ff-4375-9d98-7cdb1a9675d8" (UID: "0eb99ce9-22ff-4375-9d98-7cdb1a9675d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 21:55:51.571457 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.571426    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:51.571457 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.571455    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:51.571641 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.571466    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vn8x\" (UniqueName: \"kubernetes.io/projected/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-kube-api-access-6vn8x\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:51.571641 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.571474    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:51.865354 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.865270    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:51.870433 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:51.870404    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:52.023389 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:52.023353    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"]
Oct 29 21:55:52.026045 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:52.025856    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh" podUID="10a9a83c-2b74-4cba-8011-63a7f4f2f134" containerName="istio-proxy" containerID="cri-o://b6d825ce4356eeb4e2ff4b66ef44bf6d0e0147ac6e833ad0797b3d49dec9e487" gracePeriod=30
Oct 29 21:55:52.096831 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:52.096797    4962 generic.go:356] "Generic (PLEG): container finished" podID="0eb99ce9-22ff-4375-9d98-7cdb1a9675d8" containerID="c1d9f8375a0ccdc1a1e452f154d496dbdbc2b99ef45398318be05d4aef6beaa8" exitCode=0
Oct 29 21:55:52.097223 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:52.096839    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2"
Oct 29 21:55:52.097223 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:52.096872    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2" event={"ID":"0eb99ce9-22ff-4375-9d98-7cdb1a9675d8","Type":"ContainerDied","Data":"c1d9f8375a0ccdc1a1e452f154d496dbdbc2b99ef45398318be05d4aef6beaa8"}
Oct 29 21:55:52.097223 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:52.096931    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2" event={"ID":"0eb99ce9-22ff-4375-9d98-7cdb1a9675d8","Type":"ContainerDied","Data":"14dabf8472e923f938acc948a33f62e3b6815bcc24568e4ebe67a3c0efb3fb7d"}
Oct 29 21:55:52.097223 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:52.096952    4962 scope.go:122] "RemoveContainer" containerID="c1d9f8375a0ccdc1a1e452f154d496dbdbc2b99ef45398318be05d4aef6beaa8"
Oct 29 21:55:52.097607 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:52.097588    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:52.098655 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:52.098635    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:55:52.106786 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:52.106768    4962 scope.go:122] "RemoveContainer" containerID="c1d9f8375a0ccdc1a1e452f154d496dbdbc2b99ef45398318be05d4aef6beaa8"
Oct 29 21:55:52.107059 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:55:52.107042    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d9f8375a0ccdc1a1e452f154d496dbdbc2b99ef45398318be05d4aef6beaa8\": container with ID starting with c1d9f8375a0ccdc1a1e452f154d496dbdbc2b99ef45398318be05d4aef6beaa8 not found: ID does not exist" containerID="c1d9f8375a0ccdc1a1e452f154d496dbdbc2b99ef45398318be05d4aef6beaa8"
Oct 29 21:55:52.107100 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:52.107067    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d9f8375a0ccdc1a1e452f154d496dbdbc2b99ef45398318be05d4aef6beaa8"} err="failed to get container status \"c1d9f8375a0ccdc1a1e452f154d496dbdbc2b99ef45398318be05d4aef6beaa8\": rpc error: code = NotFound desc = could not find container \"c1d9f8375a0ccdc1a1e452f154d496dbdbc2b99ef45398318be05d4aef6beaa8\": container with ID starting with c1d9f8375a0ccdc1a1e452f154d496dbdbc2b99ef45398318be05d4aef6beaa8 not found: ID does not exist"
Oct 29 21:55:52.229826 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:52.229796    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2"]
Oct 29 21:55:52.233733 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:52.233705    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8fba532a-779dcb978d-qt6c2"]
Oct 29 21:55:53.057468 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:53.057435    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb99ce9-22ff-4375-9d98-7cdb1a9675d8" path="/var/lib/kubelet/pods/0eb99ce9-22ff-4375-9d98-7cdb1a9675d8/volumes"
Oct 29 21:55:54.601807 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:54.601767    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z"]
Oct 29 21:55:54.602231 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:54.602217    4962 cpu_manager.go:414] "containerMap: removing container" podUID="0eb99ce9-22ff-4375-9d98-7cdb1a9675d8" containerName="maas-api"
Oct 29 21:55:54.602282 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:54.602231    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb99ce9-22ff-4375-9d98-7cdb1a9675d8" containerName="maas-api"
Oct 29 21:55:54.602352 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:54.602342    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb99ce9-22ff-4375-9d98-7cdb1a9675d8" containerName="maas-api"
Oct 29 21:55:54.606820 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:54.606804    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z"
Oct 29 21:55:54.627393 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:54.627366    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z"]
Oct 29 21:55:54.694310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:54.687169    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-a9def7\""
Oct 29 21:55:54.694310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:54.687774    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-a9def7-dockercfg-kpx24\""
Oct 29 21:55:54.779380 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:54.779343    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b64e9d74-639d-44be-bdd1-04c9ef4f81fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z\" (UID: \"b64e9d74-639d-44be-bdd1-04c9ef4f81fc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z"
Oct 29 21:55:54.779606 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:54.779510    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcs7k\" (UniqueName: \"kubernetes.io/projected/b64e9d74-639d-44be-bdd1-04c9ef4f81fc-kube-api-access-rcs7k\") pod \"payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z\" (UID: \"b64e9d74-639d-44be-bdd1-04c9ef4f81fc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z"
Oct 29 21:55:54.880245 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:54.880169    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcs7k\" (UniqueName: \"kubernetes.io/projected/b64e9d74-639d-44be-bdd1-04c9ef4f81fc-kube-api-access-rcs7k\") pod \"payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z\" (UID: \"b64e9d74-639d-44be-bdd1-04c9ef4f81fc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z"
Oct 29 21:55:54.880245 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:54.880232    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b64e9d74-639d-44be-bdd1-04c9ef4f81fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z\" (UID: \"b64e9d74-639d-44be-bdd1-04c9ef4f81fc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z"
Oct 29 21:55:54.880952 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:54.880933    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b64e9d74-639d-44be-bdd1-04c9ef4f81fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z\" (UID: \"b64e9d74-639d-44be-bdd1-04c9ef4f81fc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z"
Oct 29 21:55:54.904118 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:54.904092    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcs7k\" (UniqueName: \"kubernetes.io/projected/b64e9d74-639d-44be-bdd1-04c9ef4f81fc-kube-api-access-rcs7k\") pod \"payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z\" (UID: \"b64e9d74-639d-44be-bdd1-04c9ef4f81fc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z"
Oct 29 21:55:54.919967 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:54.919947    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z"
Oct 29 21:55:55.071645 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:55.071615    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq"
Oct 29 21:55:55.224612 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:55.223513    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z"]
Oct 29 21:55:55.226687 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:55:55.226660    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64e9d74_639d_44be_bdd1_04c9ef4f81fc.slice/crio-19422e77c04e438b39c1f39fd083f0585d37db3f950c566b99082b1996ca5f40 WatchSource:0}: Error finding container 19422e77c04e438b39c1f39fd083f0585d37db3f950c566b99082b1996ca5f40: Status 404 returned error can't find the container with id 19422e77c04e438b39c1f39fd083f0585d37db3f950c566b99082b1996ca5f40
Oct 29 21:55:55.489279 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:55.489207    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd"]
Oct 29 21:55:55.493016 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:55.492997    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd"
Oct 29 21:55:55.513112 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:55.513084    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-b31f3f-serving-cert\""
Oct 29 21:55:55.519242 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:55.519219    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd"]
Oct 29 21:55:55.688258 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:55.688227    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f84b8805-aa71-45c6-b053-526ac3787a08-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd\" (UID: \"f84b8805-aa71-45c6-b053-526ac3787a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd"
Oct 29 21:55:55.688677 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:55.688279    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f84b8805-aa71-45c6-b053-526ac3787a08-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd\" (UID: \"f84b8805-aa71-45c6-b053-526ac3787a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd"
Oct 29 21:55:55.688677 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:55.688428    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f84b8805-aa71-45c6-b053-526ac3787a08-metrics-certs\") pod \"maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd\" (UID: \"f84b8805-aa71-45c6-b053-526ac3787a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd"
Oct 29 21:55:55.688677 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:55.688472    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6bcv\" (UniqueName: \"kubernetes.io/projected/f84b8805-aa71-45c6-b053-526ac3787a08-kube-api-access-l6bcv\") pod \"maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd\" (UID: \"f84b8805-aa71-45c6-b053-526ac3787a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd"
Oct 29 21:55:55.792094 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:55.792004    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f84b8805-aa71-45c6-b053-526ac3787a08-metrics-certs\") pod \"maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd\" (UID: \"f84b8805-aa71-45c6-b053-526ac3787a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd"
Oct 29 21:55:55.792094 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:55.792061    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6bcv\" (UniqueName: \"kubernetes.io/projected/f84b8805-aa71-45c6-b053-526ac3787a08-kube-api-access-l6bcv\") pod \"maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd\" (UID: \"f84b8805-aa71-45c6-b053-526ac3787a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd"
Oct 29 21:55:55.792350 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:55.792106    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f84b8805-aa71-45c6-b053-526ac3787a08-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd\" (UID: \"f84b8805-aa71-45c6-b053-526ac3787a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd"
Oct 29 21:55:55.792350 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:55.792159    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f84b8805-aa71-45c6-b053-526ac3787a08-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd\" (UID: \"f84b8805-aa71-45c6-b053-526ac3787a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd"
Oct 29 21:55:55.792920 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:55.792875    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f84b8805-aa71-45c6-b053-526ac3787a08-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd\" (UID: \"f84b8805-aa71-45c6-b053-526ac3787a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd"
Oct 29 21:55:55.795610 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:55.795586    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f84b8805-aa71-45c6-b053-526ac3787a08-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd\" (UID: \"f84b8805-aa71-45c6-b053-526ac3787a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd"
Oct 29 21:55:55.796023 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:55.795998    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f84b8805-aa71-45c6-b053-526ac3787a08-metrics-certs\") pod \"maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd\" (UID: \"f84b8805-aa71-45c6-b053-526ac3787a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd"
Oct 29 21:55:55.816784 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:55.816757    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6bcv\" (UniqueName: \"kubernetes.io/projected/f84b8805-aa71-45c6-b053-526ac3787a08-kube-api-access-l6bcv\") pod \"maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd\" (UID: \"f84b8805-aa71-45c6-b053-526ac3787a08\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd"
Oct 29 21:55:56.071576 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:56.071495    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr"
Oct 29 21:55:56.111167 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:56.111131    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd"
Oct 29 21:55:56.128327 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:56.126543    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z" event={"ID":"b64e9d74-639d-44be-bdd1-04c9ef4f81fc","Type":"ContainerStarted","Data":"c6190725a37b65115a3918d8f4bedf712e96ac58f884f2958fe990d0ee9dfaa5"}
Oct 29 21:55:56.128327 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:56.126608    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z" event={"ID":"b64e9d74-639d-44be-bdd1-04c9ef4f81fc","Type":"ContainerStarted","Data":"19422e77c04e438b39c1f39fd083f0585d37db3f950c566b99082b1996ca5f40"}
Oct 29 21:55:56.128327 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:56.126658    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z"
Oct 29 21:55:56.433650 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:56.432652    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z" podStartSLOduration=2.432632092 podStartE2EDuration="2.432632092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:56.169580562 +0000 UTC m=+577.242537526" watchObservedRunningTime="2026-10-29 21:55:56.432632092 +0000 UTC m=+577.505589059"
Oct 29 21:55:56.435877 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:56.434142    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd"]
Oct 29 21:55:56.448059 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:55:56.448026    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84b8805_aa71_45c6_b053_526ac3787a08.slice/crio-e16edc60d54579110209728c34b709b73c714b459586227bf2e539ddec4d5f31 WatchSource:0}: Error finding container e16edc60d54579110209728c34b709b73c714b459586227bf2e539ddec4d5f31: Status 404 returned error can't find the container with id e16edc60d54579110209728c34b709b73c714b459586227bf2e539ddec4d5f31
Oct 29 21:55:57.102233 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.102175    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl"
Oct 29 21:55:57.135191 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.135162    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd" event={"ID":"f84b8805-aa71-45c6-b053-526ac3787a08","Type":"ContainerStarted","Data":"f797c0ddf2aedb813cf70c2158ae0eb600dca7d6f3d43ede627228215898c3ab"}
Oct 29 21:55:57.135605 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.135577    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd"
Oct 29 21:55:57.135753 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.135622    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd" event={"ID":"f84b8805-aa71-45c6-b053-526ac3787a08","Type":"ContainerStarted","Data":"e16edc60d54579110209728c34b709b73c714b459586227bf2e539ddec4d5f31"}
Oct 29 21:55:57.209320 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.207307    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd" podStartSLOduration=1.8274712119999998 podStartE2EDuration="2.207270188s" totalImagesPullingTime="379.798976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:55:57.206376655 +0000 UTC m=+578.279333653" watchObservedRunningTime="2026-10-29 21:55:57.207270188 +0000 UTC m=+578.280227155"
Oct 29 21:55:57.451884 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.451860    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:55:57.462728 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.462703    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10a9a83c-2b74-4cba-8011-63a7f4f2f134-kube-api-access-46cqj\" (UniqueName: \"kubernetes.io/projected/10a9a83c-2b74-4cba-8011-63a7f4f2f134-kube-api-access-46cqj\") pod \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") "
Oct 29 21:55:57.462865 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.462737    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-workload-certs\") pod \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") "
Oct 29 21:55:57.462865 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.462792    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istiod-ca-cert\") pod \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") "
Oct 29 21:55:57.462865 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.462857    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-workload-socket\") pod \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") "
Oct 29 21:55:57.463024 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.462885    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-podinfo\") pod \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") "
Oct 29 21:55:57.463024 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.462914    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-data\") pod \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") "
Oct 29 21:55:57.463024 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.462974    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-token\" (UniqueName: \"kubernetes.io/projected/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-token\") pod \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") "
Oct 29 21:55:57.463175 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.463029    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-credential-socket\") pod \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") "
Oct 29 21:55:57.463175 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.463076    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-envoy\") pod \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\" (UID: \"10a9a83c-2b74-4cba-8011-63a7f4f2f134\") "
Oct 29 21:55:57.463175 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.463131    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-workload-socket" pod "10a9a83c-2b74-4cba-8011-63a7f4f2f134" (UID: "10a9a83c-2b74-4cba-8011-63a7f4f2f134"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:57.463345 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.463163    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-workload-certs" pod "10a9a83c-2b74-4cba-8011-63a7f4f2f134" (UID: "10a9a83c-2b74-4cba-8011-63a7f4f2f134"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:57.463345 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.463216    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-data" pod "10a9a83c-2b74-4cba-8011-63a7f4f2f134" (UID: "10a9a83c-2b74-4cba-8011-63a7f4f2f134"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:57.463345 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.463230    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istiod-ca-cert" pod "10a9a83c-2b74-4cba-8011-63a7f4f2f134" (UID: "10a9a83c-2b74-4cba-8011-63a7f4f2f134"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 21:55:57.463498 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.463421    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-credential-socket" pod "10a9a83c-2b74-4cba-8011-63a7f4f2f134" (UID: "10a9a83c-2b74-4cba-8011-63a7f4f2f134"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:57.463552 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.463537    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:57.463613 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.463559    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:57.463613 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.463575    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:57.463613 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.463589    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:57.463613 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.463601    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:57.464995 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.464971    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a9a83c-2b74-4cba-8011-63a7f4f2f134-kube-api-access-46cqj" pod "10a9a83c-2b74-4cba-8011-63a7f4f2f134" (UID: "10a9a83c-2b74-4cba-8011-63a7f4f2f134"). InnerVolumeSpecName "kube-api-access-46cqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:57.465140 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.465124    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-envoy" pod "10a9a83c-2b74-4cba-8011-63a7f4f2f134" (UID: "10a9a83c-2b74-4cba-8011-63a7f4f2f134"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 21:55:57.465504 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.465488    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-token" pod "10a9a83c-2b74-4cba-8011-63a7f4f2f134" (UID: "10a9a83c-2b74-4cba-8011-63a7f4f2f134"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 21:55:57.465569 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.465522    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-podinfo" pod "10a9a83c-2b74-4cba-8011-63a7f4f2f134" (UID: "10a9a83c-2b74-4cba-8011-63a7f4f2f134"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 21:55:57.566074 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.566023    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:57.566074 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.566063    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:57.566074 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.566077    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10a9a83c-2b74-4cba-8011-63a7f4f2f134-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:57.566437 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:57.566093    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46cqj\" (UniqueName: \"kubernetes.io/projected/10a9a83c-2b74-4cba-8011-63a7f4f2f134-kube-api-access-46cqj\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 21:55:58.142907 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:58.142870    4962 generic.go:356] "Generic (PLEG): container finished" podID="10a9a83c-2b74-4cba-8011-63a7f4f2f134" containerID="b6d825ce4356eeb4e2ff4b66ef44bf6d0e0147ac6e833ad0797b3d49dec9e487" exitCode=0
Oct 29 21:55:58.142907 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:58.142911    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"
Oct 29 21:55:58.143415 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:58.142950    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh" event={"ID":"10a9a83c-2b74-4cba-8011-63a7f4f2f134","Type":"ContainerDied","Data":"b6d825ce4356eeb4e2ff4b66ef44bf6d0e0147ac6e833ad0797b3d49dec9e487"}
Oct 29 21:55:58.143415 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:58.143002    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh" event={"ID":"10a9a83c-2b74-4cba-8011-63a7f4f2f134","Type":"ContainerDied","Data":"512ab4a5629739dc0d10c781e4ebe4b0a41f5d9e8b7d391f52cae51602ea1a69"}
Oct 29 21:55:58.143415 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:58.143021    4962 scope.go:122] "RemoveContainer" containerID="b6d825ce4356eeb4e2ff4b66ef44bf6d0e0147ac6e833ad0797b3d49dec9e487"
Oct 29 21:55:58.153513 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:58.153347    4962 scope.go:122] "RemoveContainer" containerID="42d21d24e8726835a75e170c7a68ea46b03430e7e24ddfe6a2fce7c459e71fa8"
Oct 29 21:55:58.162084 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:58.162062    4962 scope.go:122] "RemoveContainer" containerID="b6d825ce4356eeb4e2ff4b66ef44bf6d0e0147ac6e833ad0797b3d49dec9e487"
Oct 29 21:55:58.162371 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:55:58.162352    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6d825ce4356eeb4e2ff4b66ef44bf6d0e0147ac6e833ad0797b3d49dec9e487\": container with ID starting with b6d825ce4356eeb4e2ff4b66ef44bf6d0e0147ac6e833ad0797b3d49dec9e487 not found: ID does not exist" containerID="b6d825ce4356eeb4e2ff4b66ef44bf6d0e0147ac6e833ad0797b3d49dec9e487"
Oct 29 21:55:58.162433 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:58.162382    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6d825ce4356eeb4e2ff4b66ef44bf6d0e0147ac6e833ad0797b3d49dec9e487"} err="failed to get container status \"b6d825ce4356eeb4e2ff4b66ef44bf6d0e0147ac6e833ad0797b3d49dec9e487\": rpc error: code = NotFound desc = could not find container \"b6d825ce4356eeb4e2ff4b66ef44bf6d0e0147ac6e833ad0797b3d49dec9e487\": container with ID starting with b6d825ce4356eeb4e2ff4b66ef44bf6d0e0147ac6e833ad0797b3d49dec9e487 not found: ID does not exist"
Oct 29 21:55:58.162433 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:58.162405    4962 scope.go:122] "RemoveContainer" containerID="42d21d24e8726835a75e170c7a68ea46b03430e7e24ddfe6a2fce7c459e71fa8"
Oct 29 21:55:58.162708 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:55:58.162691    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42d21d24e8726835a75e170c7a68ea46b03430e7e24ddfe6a2fce7c459e71fa8\": container with ID starting with 42d21d24e8726835a75e170c7a68ea46b03430e7e24ddfe6a2fce7c459e71fa8 not found: ID does not exist" containerID="42d21d24e8726835a75e170c7a68ea46b03430e7e24ddfe6a2fce7c459e71fa8"
Oct 29 21:55:58.162764 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:58.162715    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42d21d24e8726835a75e170c7a68ea46b03430e7e24ddfe6a2fce7c459e71fa8"} err="failed to get container status \"42d21d24e8726835a75e170c7a68ea46b03430e7e24ddfe6a2fce7c459e71fa8\": rpc error: code = NotFound desc = could not find container \"42d21d24e8726835a75e170c7a68ea46b03430e7e24ddfe6a2fce7c459e71fa8\": container with ID starting with 42d21d24e8726835a75e170c7a68ea46b03430e7e24ddfe6a2fce7c459e71fa8 not found: ID does not exist"
Oct 29 21:55:58.208574 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:58.208550    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"]
Oct 29 21:55:58.211873 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:58.211848    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-8fba532a-gw-openshift-default-5f4f7b5d85-mlpgh"]
Oct 29 21:55:59.056808 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.056771    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10a9a83c-2b74-4cba-8011-63a7f4f2f134" path="/var/lib/kubelet/pods/10a9a83c-2b74-4cba-8011-63a7f4f2f134/volumes"
Oct 29 21:55:59.477965 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.477934    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd"]
Oct 29 21:55:59.478450 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.478435    4962 cpu_manager.go:414] "containerMap: removing container" podUID="10a9a83c-2b74-4cba-8011-63a7f4f2f134" containerName="istio-proxy"
Oct 29 21:55:59.478508 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.478452    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10a9a83c-2b74-4cba-8011-63a7f4f2f134" containerName="istio-proxy"
Oct 29 21:55:59.478508 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.478473    4962 cpu_manager.go:414] "containerMap: removing container" podUID="10a9a83c-2b74-4cba-8011-63a7f4f2f134" containerName="istio-proxy"
Oct 29 21:55:59.478508 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.478478    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10a9a83c-2b74-4cba-8011-63a7f4f2f134" containerName="istio-proxy"
Oct 29 21:55:59.478597 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.478545    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="10a9a83c-2b74-4cba-8011-63a7f4f2f134" containerName="istio-proxy"
Oct 29 21:55:59.478597 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.478556    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="10a9a83c-2b74-4cba-8011-63a7f4f2f134" containerName="istio-proxy"
Oct 29 21:55:59.483180 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.483159    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd"
Oct 29 21:55:59.497846 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.497829    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-8a9e5f-dockercfg-hl2p9\""
Oct 29 21:55:59.509556 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.509528    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-8a9e5f\""
Oct 29 21:55:59.511569 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.511546    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd"]
Oct 29 21:55:59.606947 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.606895    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee4891f-98dd-4ea8-a470-0f85ea62b849-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd\" (UID: \"cee4891f-98dd-4ea8-a470-0f85ea62b849\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd"
Oct 29 21:55:59.607120 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.607102    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bsbz\" (UniqueName: \"kubernetes.io/projected/cee4891f-98dd-4ea8-a470-0f85ea62b849-kube-api-access-4bsbz\") pod \"payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd\" (UID: \"cee4891f-98dd-4ea8-a470-0f85ea62b849\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd"
Oct 29 21:55:59.611397 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.607367    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr"]
Oct 29 21:55:59.620456 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.618915    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr"
Oct 29 21:55:59.620456 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.618811    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr"]
Oct 29 21:55:59.708080 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.708052    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bsbz\" (UniqueName: \"kubernetes.io/projected/cee4891f-98dd-4ea8-a470-0f85ea62b849-kube-api-access-4bsbz\") pod \"payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd\" (UID: \"cee4891f-98dd-4ea8-a470-0f85ea62b849\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd"
Oct 29 21:55:59.708224 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.708100    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee4891f-98dd-4ea8-a470-0f85ea62b849-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd\" (UID: \"cee4891f-98dd-4ea8-a470-0f85ea62b849\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd"
Oct 29 21:55:59.708773 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.708754    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee4891f-98dd-4ea8-a470-0f85ea62b849-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd\" (UID: \"cee4891f-98dd-4ea8-a470-0f85ea62b849\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd"
Oct 29 21:55:59.742611 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.742553    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bsbz\" (UniqueName: \"kubernetes.io/projected/cee4891f-98dd-4ea8-a470-0f85ea62b849-kube-api-access-4bsbz\") pod \"payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd\" (UID: \"cee4891f-98dd-4ea8-a470-0f85ea62b849\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd"
Oct 29 21:55:59.796225 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.796198    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd"
Oct 29 21:55:59.809666 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.809643    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37a81f7-dff2-429d-8dd1-c5d626299df8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr\" (UID: \"d37a81f7-dff2-429d-8dd1-c5d626299df8\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr"
Oct 29 21:55:59.809763 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.809713    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zshv9\" (UniqueName: \"kubernetes.io/projected/d37a81f7-dff2-429d-8dd1-c5d626299df8-kube-api-access-zshv9\") pod \"payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr\" (UID: \"d37a81f7-dff2-429d-8dd1-c5d626299df8\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr"
Oct 29 21:55:59.910748 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.910721    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37a81f7-dff2-429d-8dd1-c5d626299df8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr\" (UID: \"d37a81f7-dff2-429d-8dd1-c5d626299df8\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr"
Oct 29 21:55:59.910871 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.910779    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zshv9\" (UniqueName: \"kubernetes.io/projected/d37a81f7-dff2-429d-8dd1-c5d626299df8-kube-api-access-zshv9\") pod \"payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr\" (UID: \"d37a81f7-dff2-429d-8dd1-c5d626299df8\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr"
Oct 29 21:55:59.911406 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.911388    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37a81f7-dff2-429d-8dd1-c5d626299df8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr\" (UID: \"d37a81f7-dff2-429d-8dd1-c5d626299df8\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr"
Oct 29 21:55:59.933741 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.933715    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zshv9\" (UniqueName: \"kubernetes.io/projected/d37a81f7-dff2-429d-8dd1-c5d626299df8-kube-api-access-zshv9\") pod \"payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr\" (UID: \"d37a81f7-dff2-429d-8dd1-c5d626299df8\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr"
Oct 29 21:55:59.938579 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:55:59.938559    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr"
Oct 29 21:56:00.095461 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:00.095360    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd"]
Oct 29 21:56:00.097681 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:56:00.097654    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee4891f_98dd_4ea8_a470_0f85ea62b849.slice/crio-4a072a9f889cd09df3f647ed40dfba29e8ba4df4f5b771dc7962a105bfac1b30 WatchSource:0}: Error finding container 4a072a9f889cd09df3f647ed40dfba29e8ba4df4f5b771dc7962a105bfac1b30: Status 404 returned error can't find the container with id 4a072a9f889cd09df3f647ed40dfba29e8ba4df4f5b771dc7962a105bfac1b30
Oct 29 21:56:00.155925 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:00.155899    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd" event={"ID":"cee4891f-98dd-4ea8-a470-0f85ea62b849","Type":"ContainerStarted","Data":"4a072a9f889cd09df3f647ed40dfba29e8ba4df4f5b771dc7962a105bfac1b30"}
Oct 29 21:56:00.235123 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:00.235100    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr"]
Oct 29 21:56:00.236894 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:56:00.236867    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37a81f7_dff2_429d_8dd1_c5d626299df8.slice/crio-fc68b1d9164dbdaa5eaa70a299266cd4c2a1e48ceeb12124ecfd82620fda616c WatchSource:0}: Error finding container fc68b1d9164dbdaa5eaa70a299266cd4c2a1e48ceeb12124ecfd82620fda616c: Status 404 returned error can't find the container with id fc68b1d9164dbdaa5eaa70a299266cd4c2a1e48ceeb12124ecfd82620fda616c
Oct 29 21:56:01.122307 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.108933    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq"]
Oct 29 21:56:01.161764 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.161734    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq"]
Oct 29 21:56:01.162061 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.162046    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq"
Oct 29 21:56:01.169784 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.169755    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr" event={"ID":"d37a81f7-dff2-429d-8dd1-c5d626299df8","Type":"ContainerStarted","Data":"4092a0b64316186a01c4436afa0756c378079bc99d5911cd76886087054c1c2d"}
Oct 29 21:56:01.169967 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.169948    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr" event={"ID":"d37a81f7-dff2-429d-8dd1-c5d626299df8","Type":"ContainerStarted","Data":"fc68b1d9164dbdaa5eaa70a299266cd4c2a1e48ceeb12124ecfd82620fda616c"}
Oct 29 21:56:01.170741 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.170719    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr"
Oct 29 21:56:01.174315 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.173306    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd" event={"ID":"cee4891f-98dd-4ea8-a470-0f85ea62b849","Type":"ContainerStarted","Data":"d541e0eb15d76e65530d3b4835498b30cd423ad44a84e9c1cfd1d1f2e017be85"}
Oct 29 21:56:01.174315 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.173870    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd"
Oct 29 21:56:01.180097 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.180081    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-a9def7-serving-cert\""
Oct 29 21:56:01.248326 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.248264    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr" podStartSLOduration=2.248248335 podStartE2EDuration="2.248248335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:01.247265617 +0000 UTC m=+582.320222582" watchObservedRunningTime="2026-10-29 21:56:01.248248335 +0000 UTC m=+582.321205299"
Oct 29 21:56:01.289839 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.289786    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd" podStartSLOduration=2.289773495 podStartE2EDuration="2.289773495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:01.288158868 +0000 UTC m=+582.361115832" watchObservedRunningTime="2026-10-29 21:56:01.289773495 +0000 UTC m=+582.362730458"
Oct 29 21:56:01.325334 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.325301    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7224ed4a-e124-423c-b518-35c09cbc4383-maas-api-tls\") pod \"maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq\" (UID: \"7224ed4a-e124-423c-b518-35c09cbc4383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq"
Oct 29 21:56:01.325478 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.325402    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7224ed4a-e124-423c-b518-35c09cbc4383-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq\" (UID: \"7224ed4a-e124-423c-b518-35c09cbc4383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq"
Oct 29 21:56:01.325523 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.325471    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q4fr\" (UniqueName: \"kubernetes.io/projected/7224ed4a-e124-423c-b518-35c09cbc4383-kube-api-access-2q4fr\") pod \"maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq\" (UID: \"7224ed4a-e124-423c-b518-35c09cbc4383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq"
Oct 29 21:56:01.325788 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.325768    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7224ed4a-e124-423c-b518-35c09cbc4383-metrics-certs\") pod \"maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq\" (UID: \"7224ed4a-e124-423c-b518-35c09cbc4383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq"
Oct 29 21:56:01.430317 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.427651    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7224ed4a-e124-423c-b518-35c09cbc4383-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq\" (UID: \"7224ed4a-e124-423c-b518-35c09cbc4383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq"
Oct 29 21:56:01.430317 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.427714    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q4fr\" (UniqueName: \"kubernetes.io/projected/7224ed4a-e124-423c-b518-35c09cbc4383-kube-api-access-2q4fr\") pod \"maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq\" (UID: \"7224ed4a-e124-423c-b518-35c09cbc4383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq"
Oct 29 21:56:01.430317 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.427771    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7224ed4a-e124-423c-b518-35c09cbc4383-metrics-certs\") pod \"maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq\" (UID: \"7224ed4a-e124-423c-b518-35c09cbc4383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq"
Oct 29 21:56:01.430317 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.427905    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7224ed4a-e124-423c-b518-35c09cbc4383-maas-api-tls\") pod \"maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq\" (UID: \"7224ed4a-e124-423c-b518-35c09cbc4383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq"
Oct 29 21:56:01.430317 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.429121    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7224ed4a-e124-423c-b518-35c09cbc4383-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq\" (UID: \"7224ed4a-e124-423c-b518-35c09cbc4383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq"
Oct 29 21:56:01.447319 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.444543    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7224ed4a-e124-423c-b518-35c09cbc4383-metrics-certs\") pod \"maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq\" (UID: \"7224ed4a-e124-423c-b518-35c09cbc4383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq"
Oct 29 21:56:01.459024 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.458966    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7224ed4a-e124-423c-b518-35c09cbc4383-maas-api-tls\") pod \"maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq\" (UID: \"7224ed4a-e124-423c-b518-35c09cbc4383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq"
Oct 29 21:56:01.460376 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.459518    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q4fr\" (UniqueName: \"kubernetes.io/projected/7224ed4a-e124-423c-b518-35c09cbc4383-kube-api-access-2q4fr\") pod \"maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq\" (UID: \"7224ed4a-e124-423c-b518-35c09cbc4383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq"
Oct 29 21:56:01.487041 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.487009    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq"
Oct 29 21:56:01.800982 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:01.800959    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq"]
Oct 29 21:56:01.802616 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:56:01.802583    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7224ed4a_e124_423c_b518_35c09cbc4383.slice/crio-8ee038754cb11a7acde6b8134f599cef03f24d5ffa8264166d5dfe7ae813c51d WatchSource:0}: Error finding container 8ee038754cb11a7acde6b8134f599cef03f24d5ffa8264166d5dfe7ae813c51d: Status 404 returned error can't find the container with id 8ee038754cb11a7acde6b8134f599cef03f24d5ffa8264166d5dfe7ae813c51d
Oct 29 21:56:02.180373 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:02.180339    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq" event={"ID":"7224ed4a-e124-423c-b518-35c09cbc4383","Type":"ContainerStarted","Data":"8ee038754cb11a7acde6b8134f599cef03f24d5ffa8264166d5dfe7ae813c51d"}
Oct 29 21:56:03.152462 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:03.152422    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd"
Oct 29 21:56:03.208115 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:03.208074    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq" event={"ID":"7224ed4a-e124-423c-b518-35c09cbc4383","Type":"ContainerStarted","Data":"7faf39cc8dd034073f098cb73b95e38193d1f138c34a82c409b333c5250d1dcb"}
Oct 29 21:56:03.208649 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:03.208209    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq"
Oct 29 21:56:07.138359 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:07.138331    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z"
Oct 29 21:56:07.193312 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:07.183620    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq" podStartSLOduration=5.776819862 podStartE2EDuration="6.183602186s" totalImagesPullingTime="406.782324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:03.251345022 +0000 UTC m=+584.324302008" watchObservedRunningTime="2026-10-29 21:56:07.183602186 +0000 UTC m=+588.256559154"
Oct 29 21:56:09.219811 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:09.219783    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq"
Oct 29 21:56:09.723089 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:56:09.723032    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:56:12.288234 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:12.288198    4962 generic.go:356] "Generic (PLEG): container finished" podID="75cf3e4f-0488-4210-ab72-17b7afdeda6a" containerID="09a8d0a86523179a06b9f2864e8a2ed76f2894553b2d17cad788c3c674d66ab6" exitCode=0
Oct 29 21:56:12.288779 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:12.288278    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf" event={"ID":"75cf3e4f-0488-4210-ab72-17b7afdeda6a","Type":"ContainerDied","Data":"09a8d0a86523179a06b9f2864e8a2ed76f2894553b2d17cad788c3c674d66ab6"}
Oct 29 21:56:12.288779 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:12.288771    4962 scope.go:122] "RemoveContainer" containerID="09a8d0a86523179a06b9f2864e8a2ed76f2894553b2d17cad788c3c674d66ab6"
Oct 29 21:56:12.290356 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:12.290326    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:56:12.290480 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:12.290398    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:56:12.290480 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:12.290434    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:56:13.211219 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:13.211192    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr"
Oct 29 21:56:13.211417 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:13.211259    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd"
Oct 29 21:56:13.309537 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:13.309500    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf" event={"ID":"75cf3e4f-0488-4210-ab72-17b7afdeda6a","Type":"ContainerStarted","Data":"446ddf4d2e4b52607bffde5137ed43ae0be0a4ec18d39c786d6b3cf653919e94"}
Oct 29 21:56:13.625363 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:13.624629    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:56:13.641462 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:13.626809    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 29 21:56:13.641462 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:13.626868    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf" podUID="75cf3e4f-0488-4210-ab72-17b7afdeda6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 29 21:56:14.623813 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:14.623775    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 29 21:56:14.624442 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:14.624411    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf" podUID="75cf3e4f-0488-4210-ab72-17b7afdeda6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 29 21:56:15.627718 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:15.627679    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 29 21:56:15.628358 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:15.628328    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf" podUID="75cf3e4f-0488-4210-ab72-17b7afdeda6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 29 21:56:16.620757 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:16.620718    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 29 21:56:16.621058 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:16.621023    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf" podUID="75cf3e4f-0488-4210-ab72-17b7afdeda6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 29 21:56:17.084311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:17.083205    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:56:17.624479 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:17.624443    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 29 21:56:17.624751 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:17.624723    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf" podUID="75cf3e4f-0488-4210-ab72-17b7afdeda6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 29 21:56:19.208856 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:19.208538    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:56:19.215599 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:19.215569    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 21:56:19.264909 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:19.264879    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/2.log"
Oct 29 21:56:19.265154 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:19.265140    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/2.log"
Oct 29 21:56:19.321222 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:19.321200    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74bdq_259e6b95-d64c-4101-871f-4a933d4ac38f/ovn-acl-logging/0.log"
Oct 29 21:56:19.324060 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:19.324040    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74bdq_259e6b95-d64c-4101-871f-4a933d4ac38f/ovn-acl-logging/0.log"
Oct 29 21:56:22.150365 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:22.147279    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 21:56:22.150365 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:22.147391    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv" podUID="d71de0ec-1eaa-481f-8cee-cb256e330136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:56:23.146461 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:23.146416    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 21:56:23.146648 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:23.146474    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2" podUID="03b2ca39-a1c7-45bf-8458-f23215d334b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:56:24.286226 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.286157    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"]
Oct 29 21:56:24.289967 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.289946    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.304577 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.304554    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a3c02e-openshift-default-dockercfg-tvljv\""
Oct 29 21:56:24.321139 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.321086    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"]
Oct 29 21:56:24.362269 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.362246    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.362410 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.362283    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036f71f6-61b8-469d-a080-d1ab0b45c195-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.362460 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.362419    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdvjw\" (UniqueName: \"kubernetes.io/projected/036f71f6-61b8-469d-a080-d1ab0b45c195-kube-api-access-rdvjw\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.362496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.362459    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.362541 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.362496    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.362600 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.362586    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.362659 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.362644    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.362737 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.362723    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.362771 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.362745    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.463519 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.463479    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.463706 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.463548    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.463706 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.463577    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.463706 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.463603    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.463706 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.463640    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036f71f6-61b8-469d-a080-d1ab0b45c195-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.463706 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.463691    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdvjw\" (UniqueName: \"kubernetes.io/projected/036f71f6-61b8-469d-a080-d1ab0b45c195-kube-api-access-rdvjw\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.463998 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.463738    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.463998 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.463794    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.463998 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.463860    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.464267 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.464237    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.464267 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.464253    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.464451 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.464433    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036f71f6-61b8-469d-a080-d1ab0b45c195-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.464513 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.464457    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.464728 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.464704    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.466211 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.466183    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.466445 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.466423    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.471014 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.470991    4962 generic.go:356] "Generic (PLEG): container finished" podID="03b2ca39-a1c7-45bf-8458-f23215d334b1" containerID="d54981ad8aa1ebada11c30c0ac5a4d1cf6117480f18fec0001c3f840f139970a" exitCode=0
Oct 29 21:56:24.471116 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.471058    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2" event={"ID":"03b2ca39-a1c7-45bf-8458-f23215d334b1","Type":"ContainerDied","Data":"d54981ad8aa1ebada11c30c0ac5a4d1cf6117480f18fec0001c3f840f139970a"}
Oct 29 21:56:24.471471 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.471451    4962 scope.go:122] "RemoveContainer" containerID="d54981ad8aa1ebada11c30c0ac5a4d1cf6117480f18fec0001c3f840f139970a"
Oct 29 21:56:24.472750 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.472725    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:56:24.472808 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.472791    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:56:24.472843 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.472821    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:56:24.473800 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.473776    4962 generic.go:356] "Generic (PLEG): container finished" podID="d71de0ec-1eaa-481f-8cee-cb256e330136" containerID="7d7c79aa202bea72b52ec17fff7e032f5c0c30df32d9e7ee9b73879951fd5010" exitCode=0
Oct 29 21:56:24.473886 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.473843    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv" event={"ID":"d71de0ec-1eaa-481f-8cee-cb256e330136","Type":"ContainerDied","Data":"7d7c79aa202bea72b52ec17fff7e032f5c0c30df32d9e7ee9b73879951fd5010"}
Oct 29 21:56:24.474238 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.474222    4962 scope.go:122] "RemoveContainer" containerID="7d7c79aa202bea72b52ec17fff7e032f5c0c30df32d9e7ee9b73879951fd5010"
Oct 29 21:56:24.475775 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.475752    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:56:24.475850 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.475836    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:56:24.475890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.475871    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:56:24.486663 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.486638    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.487133 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.487113    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdvjw\" (UniqueName: \"kubernetes.io/projected/036f71f6-61b8-469d-a080-d1ab0b45c195-kube-api-access-rdvjw\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.604676 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.604348    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:24.988381 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.988354    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"]
Oct 29 21:56:24.989915 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:56:24.989886    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod036f71f6_61b8_469d_a080_d1ab0b45c195.slice/crio-1b57193608247cac8cff015727aefd501911084bb8016f4ad4ba21747221e737 WatchSource:0}: Error finding container 1b57193608247cac8cff015727aefd501911084bb8016f4ad4ba21747221e737: Status 404 returned error can't find the container with id 1b57193608247cac8cff015727aefd501911084bb8016f4ad4ba21747221e737
Oct 29 21:56:24.992814 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.992781    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:56:24.992918 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.992856    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:56:24.992983 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:24.992918    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:56:25.514310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:25.512978    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2" event={"ID":"03b2ca39-a1c7-45bf-8458-f23215d334b1","Type":"ContainerStarted","Data":"d6455e4a3c4bf2a3ecc64d4e53198a939f18bdd777da43a81925856a013ca70e"}
Oct 29 21:56:25.537314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:25.537235    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc" event={"ID":"036f71f6-61b8-469d-a080-d1ab0b45c195","Type":"ContainerStarted","Data":"271b7b89784cda842fead66ac67418b72e6abd7145340569ffe392ef431c8b62"}
Oct 29 21:56:25.537507 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:25.537334    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc" event={"ID":"036f71f6-61b8-469d-a080-d1ab0b45c195","Type":"ContainerStarted","Data":"1b57193608247cac8cff015727aefd501911084bb8016f4ad4ba21747221e737"}
Oct 29 21:56:25.539561 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:25.539536    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv" event={"ID":"d71de0ec-1eaa-481f-8cee-cb256e330136","Type":"ContainerStarted","Data":"c7bbd6fb4c953b189e12f0cbdc8d2640a8a4666d5e467a6b82d2253dd3127a19"}
Oct 29 21:56:25.606437 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:25.606397    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:25.677727 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:25.677692    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:56:25.679218 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:25.679163    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc" podStartSLOduration=1.6791451290000001 podStartE2EDuration="1.679145129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:25.629785502 +0000 UTC m=+606.702742494" watchObservedRunningTime="2026-10-29 21:56:25.679145129 +0000 UTC m=+606.752102097"
Oct 29 21:56:25.699312 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:25.684404    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 29 21:56:25.699312 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:25.684469    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv" podUID="d71de0ec-1eaa-481f-8cee-cb256e330136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 29 21:56:25.869628 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:25.869542    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:56:25.870021 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:25.869936    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 29 21:56:25.870175 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:25.870154    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2" podUID="03b2ca39-a1c7-45bf-8458-f23215d334b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 29 21:56:26.616313 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:26.615360    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 29 21:56:26.616313 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:26.615422    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc" podUID="036f71f6-61b8-469d-a080-d1ab0b45c195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 29 21:56:26.674892 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:26.674855    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 29 21:56:26.675186 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:26.675153    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv" podUID="d71de0ec-1eaa-481f-8cee-cb256e330136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 29 21:56:26.864663 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:26.864631    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 29 21:56:26.864911 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:26.864880    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2" podUID="03b2ca39-a1c7-45bf-8458-f23215d334b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 29 21:56:27.609721 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:27.609680    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 29 21:56:27.609997 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:27.609971    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc" podUID="036f71f6-61b8-469d-a080-d1ab0b45c195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 29 21:56:27.675679 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:27.675645    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 29 21:56:27.676337 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:27.676305    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv" podUID="d71de0ec-1eaa-481f-8cee-cb256e330136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 29 21:56:27.864947 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:27.864855    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 29 21:56:27.864947 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:27.864932    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2" podUID="03b2ca39-a1c7-45bf-8458-f23215d334b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 29 21:56:28.612313 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:28.607654    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 29 21:56:28.612313 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:28.607715    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc" podUID="036f71f6-61b8-469d-a080-d1ab0b45c195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 29 21:56:28.677318 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:28.675764    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 29 21:56:28.677318 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:28.675839    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv" podUID="d71de0ec-1eaa-481f-8cee-cb256e330136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 29 21:56:28.865460 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:28.865373    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 29 21:56:28.865460 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:28.865438    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2" podUID="03b2ca39-a1c7-45bf-8458-f23215d334b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 29 21:56:29.607828 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:29.607788    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 29 21:56:29.608608 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:29.608574    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc" podUID="036f71f6-61b8-469d-a080-d1ab0b45c195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 29 21:56:29.675639 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:29.675605    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 29 21:56:29.675938 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:29.675910    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv" podUID="d71de0ec-1eaa-481f-8cee-cb256e330136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 29 21:56:29.864816 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:29.864723    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 29 21:56:29.865439 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:29.865408    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2" podUID="03b2ca39-a1c7-45bf-8458-f23215d334b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 29 21:56:30.613533 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:30.613066    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 29 21:56:30.613533 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:30.613130    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc" podUID="036f71f6-61b8-469d-a080-d1ab0b45c195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 29 21:56:30.673696 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:30.673657    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 29 21:56:30.673962 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:30.673933    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv" podUID="d71de0ec-1eaa-481f-8cee-cb256e330136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 29 21:56:30.864675 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:30.864601    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Oct 29 21:56:30.864864 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:30.864845    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2" podUID="03b2ca39-a1c7-45bf-8458-f23215d334b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Oct 29 21:56:31.336914 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.336868    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"]
Oct 29 21:56:31.402693 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.402663    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"]
Oct 29 21:56:31.403073 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.403054    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.462408 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.460453    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-e4e24e-openshift-default-dockercfg-9t52r\""
Oct 29 21:56:31.513339 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.504257    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.519878 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.519808    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.520147 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.520088    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.520336 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.520279    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.520429 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.520391    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.520657 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.520558    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.520657 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.520596    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.520657 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.520635    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.520851 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.520746    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwksj\" (UniqueName: \"kubernetes.io/projected/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-kube-api-access-jwksj\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.605367 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.605249    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 29 21:56:31.605367 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.605337    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc" podUID="036f71f6-61b8-469d-a080-d1ab0b45c195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 29 21:56:31.621690 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.621648    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.622438 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.622280    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.622896 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.622847    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.623020 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.622381    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.623155 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.623138    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.623695 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.623676    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.624256 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.624234    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.624837 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.624812    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.624962 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.624856    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.624962 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.624894    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.624962 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.624947    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwksj\" (UniqueName: \"kubernetes.io/projected/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-kube-api-access-jwksj\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.625795 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.625462    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.625795 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.625753    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.626019 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.625973    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.626312 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.626259    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.628271 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.628223    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.648411 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.648383    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwksj\" (UniqueName: \"kubernetes.io/projected/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-kube-api-access-jwksj\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.649129 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.649101    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:31.680021 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.679995    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:56:31.680584 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.680562    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:56:31.682563 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.681650    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 21:56:31.838387 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:31.837969    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:32.035246 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:32.035209    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:56:32.035989 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:32.035971    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:56:32.037779 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:32.037760    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 21:56:32.179571 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:32.179547    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"]
Oct 29 21:56:32.186985 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:56:32.186959    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef87fe2_429a_4393_8ddf_fa43e2dd9044.slice/crio-dd1eeaf3327a80b0c05e37a819df4750e4ff0631b0fd5bfab28f77c3ef33d6aa WatchSource:0}: Error finding container dd1eeaf3327a80b0c05e37a819df4750e4ff0631b0fd5bfab28f77c3ef33d6aa: Status 404 returned error can't find the container with id dd1eeaf3327a80b0c05e37a819df4750e4ff0631b0fd5bfab28f77c3ef33d6aa
Oct 29 21:56:32.189512 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:32.189480    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:56:32.189631 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:32.189555    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:56:32.189694 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:32.189626    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:56:32.639653 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:32.639489    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr" event={"ID":"4ef87fe2-429a-4393-8ddf-fa43e2dd9044","Type":"ContainerStarted","Data":"7cd80b4e67697abfff4dcd0c68f6f8feae551f87b263441e3db252dfde348611"}
Oct 29 21:56:32.639653 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:32.639555    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr" event={"ID":"4ef87fe2-429a-4393-8ddf-fa43e2dd9044","Type":"ContainerStarted","Data":"dd1eeaf3327a80b0c05e37a819df4750e4ff0631b0fd5bfab28f77c3ef33d6aa"}
Oct 29 21:56:32.687713 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:32.687621    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr" podStartSLOduration=1.687599682 podStartE2EDuration="1.687599682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:32.686392908 +0000 UTC m=+613.759349900" watchObservedRunningTime="2026-10-29 21:56:32.687599682 +0000 UTC m=+613.760556706"
Oct 29 21:56:32.838636 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:32.838599    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:33.129784 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:33.129752    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:33.130250 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:33.130073    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:33.166665 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:33.166643    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 21:56:33.840810 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:33.840777    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 29 21:56:33.840985 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:33.840825    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr" podUID="4ef87fe2-429a-4393-8ddf-fa43e2dd9044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 29 21:56:34.404049 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.403999    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"]
Oct 29 21:56:34.407954 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.407933    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:56:34.434890 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.434866    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a9def7\"/\"openshift-service-ca.crt\""
Oct 29 21:56:34.435146 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.435125    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a9def7\"/\"simulator-w2-a9def7-kserve-self-signed-certs\""
Oct 29 21:56:34.435228 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.435186    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a9def7\"/\"default-dockercfg-26fhs\""
Oct 29 21:56:34.435362 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.435347    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a9def7\"/\"kube-root-ca.crt\""
Oct 29 21:56:34.438140 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.438118    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"]
Oct 29 21:56:34.500644 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.500612    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-tmp-dir\") pod \"simulator-w2-a9def7-kserve-5ff46c6f67-kvldd\" (UID: \"f1ebf370-2322-49c9-8615-2d7109f57083\") " pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:56:34.500791 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.500662    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1ebf370-2322-49c9-8615-2d7109f57083-tls-certs\") pod \"simulator-w2-a9def7-kserve-5ff46c6f67-kvldd\" (UID: \"f1ebf370-2322-49c9-8615-2d7109f57083\") " pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:56:34.500832 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.500793    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-home\") pod \"simulator-w2-a9def7-kserve-5ff46c6f67-kvldd\" (UID: \"f1ebf370-2322-49c9-8615-2d7109f57083\") " pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:56:34.500881 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.500829    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-dshm\") pod \"simulator-w2-a9def7-kserve-5ff46c6f67-kvldd\" (UID: \"f1ebf370-2322-49c9-8615-2d7109f57083\") " pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:56:34.500881 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.500858    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l7pn\" (UniqueName: \"kubernetes.io/projected/f1ebf370-2322-49c9-8615-2d7109f57083-kube-api-access-7l7pn\") pod \"simulator-w2-a9def7-kserve-5ff46c6f67-kvldd\" (UID: \"f1ebf370-2322-49c9-8615-2d7109f57083\") " pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:56:34.501022 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.501006    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-model-cache\") pod \"simulator-w2-a9def7-kserve-5ff46c6f67-kvldd\" (UID: \"f1ebf370-2322-49c9-8615-2d7109f57083\") " pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:56:34.602162 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.602132    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-home\") pod \"simulator-w2-a9def7-kserve-5ff46c6f67-kvldd\" (UID: \"f1ebf370-2322-49c9-8615-2d7109f57083\") " pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:56:34.602332 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.602168    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-dshm\") pod \"simulator-w2-a9def7-kserve-5ff46c6f67-kvldd\" (UID: \"f1ebf370-2322-49c9-8615-2d7109f57083\") " pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:56:34.602332 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.602188    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l7pn\" (UniqueName: \"kubernetes.io/projected/f1ebf370-2322-49c9-8615-2d7109f57083-kube-api-access-7l7pn\") pod \"simulator-w2-a9def7-kserve-5ff46c6f67-kvldd\" (UID: \"f1ebf370-2322-49c9-8615-2d7109f57083\") " pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:56:34.602332 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.602218    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-model-cache\") pod \"simulator-w2-a9def7-kserve-5ff46c6f67-kvldd\" (UID: \"f1ebf370-2322-49c9-8615-2d7109f57083\") " pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:56:34.602501 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.602336    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-tmp-dir\") pod \"simulator-w2-a9def7-kserve-5ff46c6f67-kvldd\" (UID: \"f1ebf370-2322-49c9-8615-2d7109f57083\") " pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:56:34.602501 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.602391    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1ebf370-2322-49c9-8615-2d7109f57083-tls-certs\") pod \"simulator-w2-a9def7-kserve-5ff46c6f67-kvldd\" (UID: \"f1ebf370-2322-49c9-8615-2d7109f57083\") " pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:56:34.602704 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.602674    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-model-cache\") pod \"simulator-w2-a9def7-kserve-5ff46c6f67-kvldd\" (UID: \"f1ebf370-2322-49c9-8615-2d7109f57083\") " pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:56:34.602837 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.602735    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-home\") pod \"simulator-w2-a9def7-kserve-5ff46c6f67-kvldd\" (UID: \"f1ebf370-2322-49c9-8615-2d7109f57083\") " pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:56:34.602837 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.602739    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-tmp-dir\") pod \"simulator-w2-a9def7-kserve-5ff46c6f67-kvldd\" (UID: \"f1ebf370-2322-49c9-8615-2d7109f57083\") " pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:56:34.604495 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.604471    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-dshm\") pod \"simulator-w2-a9def7-kserve-5ff46c6f67-kvldd\" (UID: \"f1ebf370-2322-49c9-8615-2d7109f57083\") " pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:56:34.604997 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.604978    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1ebf370-2322-49c9-8615-2d7109f57083-tls-certs\") pod \"simulator-w2-a9def7-kserve-5ff46c6f67-kvldd\" (UID: \"f1ebf370-2322-49c9-8615-2d7109f57083\") " pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:56:34.625514 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.625496    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l7pn\" (UniqueName: \"kubernetes.io/projected/f1ebf370-2322-49c9-8615-2d7109f57083-kube-api-access-7l7pn\") pod \"simulator-w2-a9def7-kserve-5ff46c6f67-kvldd\" (UID: \"f1ebf370-2322-49c9-8615-2d7109f57083\") " pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:56:34.728853 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.728501    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:56:34.840785 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.840242    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 29 21:56:34.840785 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:34.840347    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr" podUID="4ef87fe2-429a-4393-8ddf-fa43e2dd9044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 29 21:56:35.066268 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:35.066243    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"]
Oct 29 21:56:35.067723 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:56:35.067699    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1ebf370_2322_49c9_8615_2d7109f57083.slice/crio-790109bbc7971f3f1275bbfd62df875c0c35e8f64aa031164bf936cfd336e0fb WatchSource:0}: Error finding container 790109bbc7971f3f1275bbfd62df875c0c35e8f64aa031164bf936cfd336e0fb: Status 404 returned error can't find the container with id 790109bbc7971f3f1275bbfd62df875c0c35e8f64aa031164bf936cfd336e0fb
Oct 29 21:56:35.699573 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:35.699538    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" event={"ID":"f1ebf370-2322-49c9-8615-2d7109f57083","Type":"ContainerStarted","Data":"b3559de0ac25a5d498778e4412fa0d991baf71793ec3ed0a09fcda072634e633"}
Oct 29 21:56:35.699573 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:35.699580    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" event={"ID":"f1ebf370-2322-49c9-8615-2d7109f57083","Type":"ContainerStarted","Data":"790109bbc7971f3f1275bbfd62df875c0c35e8f64aa031164bf936cfd336e0fb"}
Oct 29 21:56:35.729349 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:35.729312    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:56:35.743016 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:35.742976    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podStartSLOduration=1.742964308 podStartE2EDuration="1.742964308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:35.741074006 +0000 UTC m=+616.814030971" watchObservedRunningTime="2026-10-29 21:56:35.742964308 +0000 UTC m=+616.815921272"
Oct 29 21:56:35.839204 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:35.839175    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 29 21:56:35.839354 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:35.839229    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr" podUID="4ef87fe2-429a-4393-8ddf-fa43e2dd9044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 29 21:56:36.839038 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:36.839005    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 29 21:56:36.839520 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:36.839059    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr" podUID="4ef87fe2-429a-4393-8ddf-fa43e2dd9044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 29 21:56:37.838877 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:37.838843    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 29 21:56:37.839062 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:37.838901    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr" podUID="4ef87fe2-429a-4393-8ddf-fa43e2dd9044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 29 21:56:39.102594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:39.102554    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:39.103223 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:39.103194    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:39.104179 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:39.104158    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 21:56:39.722504 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:56:39.722464    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:56:42.894918 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:42.894880    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"]
Oct 29 21:56:42.900778 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:42.900758    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:56:42.917698 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:42.917675    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8a9e5f\"/\"openshift-service-ca.crt\""
Oct 29 21:56:42.929522 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:42.929493    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8a9e5f\"/\"simulator-w3-8a9e5f-kserve-self-signed-certs\""
Oct 29 21:56:42.929522 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:42.929496    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8a9e5f\"/\"kube-root-ca.crt\""
Oct 29 21:56:42.929676 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:42.929544    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8a9e5f\"/\"default-dockercfg-q247w\""
Oct 29 21:56:42.933268 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:42.933247    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"]
Oct 29 21:56:43.017609 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:43.017582    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-model-cache\") pod \"simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v\" (UID: \"e609098d-a77f-45bd-b87b-7e579dd0ed73\") " pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:56:43.017738 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:43.017632    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-tmp-dir\") pod \"simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v\" (UID: \"e609098d-a77f-45bd-b87b-7e579dd0ed73\") " pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:56:43.017782 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:43.017764    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e609098d-a77f-45bd-b87b-7e579dd0ed73-tls-certs\") pod \"simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v\" (UID: \"e609098d-a77f-45bd-b87b-7e579dd0ed73\") " pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:56:43.017821 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:43.017792    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfvjq\" (UniqueName: \"kubernetes.io/projected/e609098d-a77f-45bd-b87b-7e579dd0ed73-kube-api-access-nfvjq\") pod \"simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v\" (UID: \"e609098d-a77f-45bd-b87b-7e579dd0ed73\") " pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:56:43.017821 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:43.017811    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-dshm\") pod \"simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v\" (UID: \"e609098d-a77f-45bd-b87b-7e579dd0ed73\") " pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:56:43.017887 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:43.017831    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-home\") pod \"simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v\" (UID: \"e609098d-a77f-45bd-b87b-7e579dd0ed73\") " pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:56:43.141798 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:43.141762    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-model-cache\") pod \"simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v\" (UID: \"e609098d-a77f-45bd-b87b-7e579dd0ed73\") " pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:56:43.141989 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:43.141851    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-tmp-dir\") pod \"simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v\" (UID: \"e609098d-a77f-45bd-b87b-7e579dd0ed73\") " pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:56:43.141989 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:43.141942    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e609098d-a77f-45bd-b87b-7e579dd0ed73-tls-certs\") pod \"simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v\" (UID: \"e609098d-a77f-45bd-b87b-7e579dd0ed73\") " pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:56:43.141989 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:43.141970    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfvjq\" (UniqueName: \"kubernetes.io/projected/e609098d-a77f-45bd-b87b-7e579dd0ed73-kube-api-access-nfvjq\") pod \"simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v\" (UID: \"e609098d-a77f-45bd-b87b-7e579dd0ed73\") " pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:56:43.142154 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:43.141998    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-dshm\") pod \"simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v\" (UID: \"e609098d-a77f-45bd-b87b-7e579dd0ed73\") " pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:56:43.142154 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:43.142029    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-home\") pod \"simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v\" (UID: \"e609098d-a77f-45bd-b87b-7e579dd0ed73\") " pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:56:43.159900 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:43.142550    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-home\") pod \"simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v\" (UID: \"e609098d-a77f-45bd-b87b-7e579dd0ed73\") " pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:56:43.159900 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:43.143024    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-model-cache\") pod \"simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v\" (UID: \"e609098d-a77f-45bd-b87b-7e579dd0ed73\") " pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:56:43.159900 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:43.143092    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-tmp-dir\") pod \"simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v\" (UID: \"e609098d-a77f-45bd-b87b-7e579dd0ed73\") " pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:56:43.159900 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:43.157527    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-dshm\") pod \"simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v\" (UID: \"e609098d-a77f-45bd-b87b-7e579dd0ed73\") " pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:56:43.159900 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:43.159719    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e609098d-a77f-45bd-b87b-7e579dd0ed73-tls-certs\") pod \"simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v\" (UID: \"e609098d-a77f-45bd-b87b-7e579dd0ed73\") " pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:56:43.194148 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:43.194056    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfvjq\" (UniqueName: \"kubernetes.io/projected/e609098d-a77f-45bd-b87b-7e579dd0ed73-kube-api-access-nfvjq\") pod \"simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v\" (UID: \"e609098d-a77f-45bd-b87b-7e579dd0ed73\") " pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:56:43.213865 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:43.212821    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:56:43.531301 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:43.531258    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"]
Oct 29 21:56:43.532174 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:56:43.532145    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode609098d_a77f_45bd_b87b_7e579dd0ed73.slice/crio-ceb1b08e912b699c61f247d3d5991ab00853317962572048a733c9717908bdcb WatchSource:0}: Error finding container ceb1b08e912b699c61f247d3d5991ab00853317962572048a733c9717908bdcb: Status 404 returned error can't find the container with id ceb1b08e912b699c61f247d3d5991ab00853317962572048a733c9717908bdcb
Oct 29 21:56:43.766941 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:43.766904    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" event={"ID":"e609098d-a77f-45bd-b87b-7e579dd0ed73","Type":"ContainerStarted","Data":"cd94d2fe13aa040aae836ba36a81875aaaef0ff3c3c4c8dcf88a7bd7e8fe5621"}
Oct 29 21:56:43.767087 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:43.766956    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" event={"ID":"e609098d-a77f-45bd-b87b-7e579dd0ed73","Type":"ContainerStarted","Data":"ceb1b08e912b699c61f247d3d5991ab00853317962572048a733c9717908bdcb"}
Oct 29 21:56:43.810494 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:43.810412    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podStartSLOduration=1.810399278 podStartE2EDuration="1.810399278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:43.808697736 +0000 UTC m=+624.881654712" watchObservedRunningTime="2026-10-29 21:56:43.810399278 +0000 UTC m=+624.883356241"
Oct 29 21:56:44.213347 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:44.213316    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:56:44.730025 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:44.729994    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:56:44.742441 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:44.742419    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:56:44.783458 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:44.783432    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:56:53.213954 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:53.213926    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:56:53.226124 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:53.226102    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:56:53.832053 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:53.832021    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:56:56.545355 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.545319    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"]
Oct 29 21:56:56.551160 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.551135    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:56:56.566121 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.566093    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"]
Oct 29 21:56:56.575131 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.575093    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b31f3f\"/\"openshift-service-ca.crt\""
Oct 29 21:56:56.575248 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.575137    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b31f3f\"/\"kube-root-ca.crt\""
Oct 29 21:56:56.577218 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.577197    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b31f3f\"/\"default-dockercfg-6tf66\""
Oct 29 21:56:56.577333 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.577226    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b31f3f\"/\"premium-w4-b31f3f-kserve-self-signed-certs\""
Oct 29 21:56:56.581432 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.581409    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2883ab31-9399-4c28-af6a-9c66fd043396-tls-certs\") pod \"premium-w4-b31f3f-kserve-66f587688f-bdc4h\" (UID: \"2883ab31-9399-4c28-af6a-9c66fd043396\") " pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:56:56.581519 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.581467    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9gqk\" (UniqueName: \"kubernetes.io/projected/2883ab31-9399-4c28-af6a-9c66fd043396-kube-api-access-k9gqk\") pod \"premium-w4-b31f3f-kserve-66f587688f-bdc4h\" (UID: \"2883ab31-9399-4c28-af6a-9c66fd043396\") " pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:56:56.581581 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.581565    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-model-cache\") pod \"premium-w4-b31f3f-kserve-66f587688f-bdc4h\" (UID: \"2883ab31-9399-4c28-af6a-9c66fd043396\") " pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:56:56.581621 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.581595    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-tmp-dir\") pod \"premium-w4-b31f3f-kserve-66f587688f-bdc4h\" (UID: \"2883ab31-9399-4c28-af6a-9c66fd043396\") " pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:56:56.581668 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.581627    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-home\") pod \"premium-w4-b31f3f-kserve-66f587688f-bdc4h\" (UID: \"2883ab31-9399-4c28-af6a-9c66fd043396\") " pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:56:56.581760 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.581744    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-dshm\") pod \"premium-w4-b31f3f-kserve-66f587688f-bdc4h\" (UID: \"2883ab31-9399-4c28-af6a-9c66fd043396\") " pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:56:56.682597 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.682566    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9gqk\" (UniqueName: \"kubernetes.io/projected/2883ab31-9399-4c28-af6a-9c66fd043396-kube-api-access-k9gqk\") pod \"premium-w4-b31f3f-kserve-66f587688f-bdc4h\" (UID: \"2883ab31-9399-4c28-af6a-9c66fd043396\") " pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:56:56.682705 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.682619    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-model-cache\") pod \"premium-w4-b31f3f-kserve-66f587688f-bdc4h\" (UID: \"2883ab31-9399-4c28-af6a-9c66fd043396\") " pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:56:56.682705 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.682654    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-tmp-dir\") pod \"premium-w4-b31f3f-kserve-66f587688f-bdc4h\" (UID: \"2883ab31-9399-4c28-af6a-9c66fd043396\") " pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:56:56.682705 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.682689    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-home\") pod \"premium-w4-b31f3f-kserve-66f587688f-bdc4h\" (UID: \"2883ab31-9399-4c28-af6a-9c66fd043396\") " pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:56:56.682862 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.682766    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-dshm\") pod \"premium-w4-b31f3f-kserve-66f587688f-bdc4h\" (UID: \"2883ab31-9399-4c28-af6a-9c66fd043396\") " pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:56:56.682862 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.682837    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2883ab31-9399-4c28-af6a-9c66fd043396-tls-certs\") pod \"premium-w4-b31f3f-kserve-66f587688f-bdc4h\" (UID: \"2883ab31-9399-4c28-af6a-9c66fd043396\") " pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:56:56.683083 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.683058    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-model-cache\") pod \"premium-w4-b31f3f-kserve-66f587688f-bdc4h\" (UID: \"2883ab31-9399-4c28-af6a-9c66fd043396\") " pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:56:56.683152 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.683103    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-tmp-dir\") pod \"premium-w4-b31f3f-kserve-66f587688f-bdc4h\" (UID: \"2883ab31-9399-4c28-af6a-9c66fd043396\") " pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:56:56.683152 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.683134    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-home\") pod \"premium-w4-b31f3f-kserve-66f587688f-bdc4h\" (UID: \"2883ab31-9399-4c28-af6a-9c66fd043396\") " pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:56:56.685034 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.685014    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-dshm\") pod \"premium-w4-b31f3f-kserve-66f587688f-bdc4h\" (UID: \"2883ab31-9399-4c28-af6a-9c66fd043396\") " pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:56:56.685309 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.685279    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2883ab31-9399-4c28-af6a-9c66fd043396-tls-certs\") pod \"premium-w4-b31f3f-kserve-66f587688f-bdc4h\" (UID: \"2883ab31-9399-4c28-af6a-9c66fd043396\") " pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:56:56.734862 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.734834    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9gqk\" (UniqueName: \"kubernetes.io/projected/2883ab31-9399-4c28-af6a-9c66fd043396-kube-api-access-k9gqk\") pod \"premium-w4-b31f3f-kserve-66f587688f-bdc4h\" (UID: \"2883ab31-9399-4c28-af6a-9c66fd043396\") " pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:56:56.866358 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:56.866046    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:56:57.194518 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:57.194493    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"]
Oct 29 21:56:57.195543 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:56:57.195518    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2883ab31_9399_4c28_af6a_9c66fd043396.slice/crio-aaba27141d62906b0c54449a27ebb0ae813698022017463cf06774695ed14327 WatchSource:0}: Error finding container aaba27141d62906b0c54449a27ebb0ae813698022017463cf06774695ed14327: Status 404 returned error can't find the container with id aaba27141d62906b0c54449a27ebb0ae813698022017463cf06774695ed14327
Oct 29 21:56:57.873721 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:57.873685    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" event={"ID":"2883ab31-9399-4c28-af6a-9c66fd043396","Type":"ContainerStarted","Data":"7dffb80a37b687e99dfa2404188d26dcf375bc193b0616d63e0f87c6fcb8810d"}
Oct 29 21:56:57.874099 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:57.873728    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" event={"ID":"2883ab31-9399-4c28-af6a-9c66fd043396","Type":"ContainerStarted","Data":"aaba27141d62906b0c54449a27ebb0ae813698022017463cf06774695ed14327"}
Oct 29 21:56:57.916980 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:57.916933    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podStartSLOduration=1.916921725 podStartE2EDuration="1.916921725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:56:57.91533177 +0000 UTC m=+638.988288735" watchObservedRunningTime="2026-10-29 21:56:57.916921725 +0000 UTC m=+638.989878689"
Oct 29 21:56:58.867303 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:56:58.867267    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:57:06.866864 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:06.866830    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:57:06.879123 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:06.879097    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:57:06.932563 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:06.932532    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:57:07.377782 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.377752    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"]
Oct 29 21:57:07.381756 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.381734    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.396390 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.396369    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-a940bc-openshift-default-dockercfg-cl8t8\""
Oct 29 21:57:07.411248 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.411221    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"]
Oct 29 21:57:07.500280 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.500251    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.500427 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.500283    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.500427 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.500374    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.500502 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.500453    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.500542 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.500503    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.500601 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.500582    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.500645 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.500615    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.500645 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.500637    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.500737 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.500724    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knkff\" (UniqueName: \"kubernetes.io/projected/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-kube-api-access-knkff\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.601190 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.601165    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.601321 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.601199    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.601321 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.601220    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.601321 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.601256    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knkff\" (UniqueName: \"kubernetes.io/projected/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-kube-api-access-knkff\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.601479 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.601323    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.601479 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.601348    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.601479 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.601368    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.601479 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.601440    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.601660 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.601481    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.601718 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.601678    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.601777 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.601757    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.601892 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.601871    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.601957 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.601905    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.602159 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.602143    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.603636 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.603614    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.603934 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.603915    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.628957 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.628898    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knkff\" (UniqueName: \"kubernetes.io/projected/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-kube-api-access-knkff\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.630632 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.630612    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:07.695603 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:07.695577    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:08.014772 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:08.014738    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"]
Oct 29 21:57:08.029061 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:57:08.029033    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72eb074_2855_4dd2_bf0b_e4a42e92d8bb.slice/crio-c675f7f724204b29e627a57d9fc5250553f253e612491d9ed83cbedb4b3d801b WatchSource:0}: Error finding container c675f7f724204b29e627a57d9fc5250553f253e612491d9ed83cbedb4b3d801b: Status 404 returned error can't find the container with id c675f7f724204b29e627a57d9fc5250553f253e612491d9ed83cbedb4b3d801b
Oct 29 21:57:08.031763 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:08.031733    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:57:08.031831 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:08.031796    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:57:08.031831 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:08.031823    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:57:08.939142 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:08.939104    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc" event={"ID":"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb","Type":"ContainerStarted","Data":"d25824e90d88ebcedeea778ec0f75210f943a5a9ab3142969b7d5f26a4de63aa"}
Oct 29 21:57:08.939372 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:08.939168    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc" event={"ID":"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb","Type":"ContainerStarted","Data":"c675f7f724204b29e627a57d9fc5250553f253e612491d9ed83cbedb4b3d801b"}
Oct 29 21:57:09.003311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:09.001761    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc" podStartSLOduration=2.001743319 podStartE2EDuration="2.001743319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:09.00080885 +0000 UTC m=+650.073765815" watchObservedRunningTime="2026-10-29 21:57:09.001743319 +0000 UTC m=+650.074700284"
Oct 29 21:57:09.700308 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:09.699655    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:09.704310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:09.701918    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 29 21:57:09.704310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:09.701982    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc" podUID="b72eb074-2855-4dd2-bf0b-e4a42e92d8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 29 21:57:09.734889 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:57:09.734857    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:57:10.703311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:10.702595    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 29 21:57:10.703311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:10.702680    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc" podUID="b72eb074-2855-4dd2-bf0b-e4a42e92d8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 29 21:57:11.696845 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:11.696810    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 29 21:57:11.697067 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:11.696875    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc" podUID="b72eb074-2855-4dd2-bf0b-e4a42e92d8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 29 21:57:12.696675 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:12.696632    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 29 21:57:12.697122 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:12.696705    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc" podUID="b72eb074-2855-4dd2-bf0b-e4a42e92d8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 29 21:57:13.702879 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:13.702400    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 29 21:57:13.702879 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:13.702469    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc" podUID="b72eb074-2855-4dd2-bf0b-e4a42e92d8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 29 21:57:14.790739 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:14.790701    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:14.791281 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:14.791106    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:14.791847 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:14.791826    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 21:57:19.073580 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.073539    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"]
Oct 29 21:57:19.086656 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.086628    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:57:19.100242 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.100220    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"]
Oct 29 21:57:19.108557 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.108533    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-743a5e\"/\"openshift-service-ca.crt\""
Oct 29 21:57:19.108675 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.108660    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-743a5e\"/\"kube-root-ca.crt\""
Oct 29 21:57:19.112891 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.112867    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-743a5e\"/\"default-dockercfg-skx5w\""
Oct 29 21:57:19.113021 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.112899    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-743a5e\"/\"premium-w5-743a5e-kserve-self-signed-certs\""
Oct 29 21:57:19.223753 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.223718    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-home\") pod \"premium-w5-743a5e-kserve-d877dd66-lq47b\" (UID: \"262b3d82-eee9-4a36-b7e3-244aa1641c07\") " pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:57:19.223918 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.223819    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdthz\" (UniqueName: \"kubernetes.io/projected/262b3d82-eee9-4a36-b7e3-244aa1641c07-kube-api-access-jdthz\") pod \"premium-w5-743a5e-kserve-d877dd66-lq47b\" (UID: \"262b3d82-eee9-4a36-b7e3-244aa1641c07\") " pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:57:19.223918 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.223857    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-model-cache\") pod \"premium-w5-743a5e-kserve-d877dd66-lq47b\" (UID: \"262b3d82-eee9-4a36-b7e3-244aa1641c07\") " pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:57:19.223918 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.223881    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-dshm\") pod \"premium-w5-743a5e-kserve-d877dd66-lq47b\" (UID: \"262b3d82-eee9-4a36-b7e3-244aa1641c07\") " pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:57:19.224082 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.223931    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-tmp-dir\") pod \"premium-w5-743a5e-kserve-d877dd66-lq47b\" (UID: \"262b3d82-eee9-4a36-b7e3-244aa1641c07\") " pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:57:19.224139 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.224101    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/262b3d82-eee9-4a36-b7e3-244aa1641c07-tls-certs\") pod \"premium-w5-743a5e-kserve-d877dd66-lq47b\" (UID: \"262b3d82-eee9-4a36-b7e3-244aa1641c07\") " pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:57:19.328267 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.325235    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-home\") pod \"premium-w5-743a5e-kserve-d877dd66-lq47b\" (UID: \"262b3d82-eee9-4a36-b7e3-244aa1641c07\") " pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:57:19.328267 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.325415    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdthz\" (UniqueName: \"kubernetes.io/projected/262b3d82-eee9-4a36-b7e3-244aa1641c07-kube-api-access-jdthz\") pod \"premium-w5-743a5e-kserve-d877dd66-lq47b\" (UID: \"262b3d82-eee9-4a36-b7e3-244aa1641c07\") " pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:57:19.328267 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.325532    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-model-cache\") pod \"premium-w5-743a5e-kserve-d877dd66-lq47b\" (UID: \"262b3d82-eee9-4a36-b7e3-244aa1641c07\") " pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:57:19.328267 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.325599    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-dshm\") pod \"premium-w5-743a5e-kserve-d877dd66-lq47b\" (UID: \"262b3d82-eee9-4a36-b7e3-244aa1641c07\") " pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:57:19.328267 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.325640    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-tmp-dir\") pod \"premium-w5-743a5e-kserve-d877dd66-lq47b\" (UID: \"262b3d82-eee9-4a36-b7e3-244aa1641c07\") " pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:57:19.328267 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.325800    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/262b3d82-eee9-4a36-b7e3-244aa1641c07-tls-certs\") pod \"premium-w5-743a5e-kserve-d877dd66-lq47b\" (UID: \"262b3d82-eee9-4a36-b7e3-244aa1641c07\") " pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:57:19.328267 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.326706    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-model-cache\") pod \"premium-w5-743a5e-kserve-d877dd66-lq47b\" (UID: \"262b3d82-eee9-4a36-b7e3-244aa1641c07\") " pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:57:19.328267 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.327000    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-home\") pod \"premium-w5-743a5e-kserve-d877dd66-lq47b\" (UID: \"262b3d82-eee9-4a36-b7e3-244aa1641c07\") " pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:57:19.328267 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.327913    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-tmp-dir\") pod \"premium-w5-743a5e-kserve-d877dd66-lq47b\" (UID: \"262b3d82-eee9-4a36-b7e3-244aa1641c07\") " pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:57:19.336715 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.336685    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-dshm\") pod \"premium-w5-743a5e-kserve-d877dd66-lq47b\" (UID: \"262b3d82-eee9-4a36-b7e3-244aa1641c07\") " pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:57:19.337323 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.337280    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/262b3d82-eee9-4a36-b7e3-244aa1641c07-tls-certs\") pod \"premium-w5-743a5e-kserve-d877dd66-lq47b\" (UID: \"262b3d82-eee9-4a36-b7e3-244aa1641c07\") " pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:57:19.351066 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.351038    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdthz\" (UniqueName: \"kubernetes.io/projected/262b3d82-eee9-4a36-b7e3-244aa1641c07-kube-api-access-jdthz\") pod \"premium-w5-743a5e-kserve-d877dd66-lq47b\" (UID: \"262b3d82-eee9-4a36-b7e3-244aa1641c07\") " pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:57:19.400266 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.400238    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:57:19.702222 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:19.702199    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"]
Oct 29 21:57:19.703748 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:57:19.703720    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod262b3d82_eee9_4a36_b7e3_244aa1641c07.slice/crio-7ae4156820245c31d3e47405f4885e40f46fbee51d4bef4a24d043167df90b9b WatchSource:0}: Error finding container 7ae4156820245c31d3e47405f4885e40f46fbee51d4bef4a24d043167df90b9b: Status 404 returned error can't find the container with id 7ae4156820245c31d3e47405f4885e40f46fbee51d4bef4a24d043167df90b9b
Oct 29 21:57:20.095934 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:20.095844    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" event={"ID":"262b3d82-eee9-4a36-b7e3-244aa1641c07","Type":"ContainerStarted","Data":"38678d15c466655bc92c1f48a898a191141840e68129c8fa0f6d70e03bbda5ad"}
Oct 29 21:57:20.095934 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:20.095884    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" event={"ID":"262b3d82-eee9-4a36-b7e3-244aa1641c07","Type":"ContainerStarted","Data":"7ae4156820245c31d3e47405f4885e40f46fbee51d4bef4a24d043167df90b9b"}
Oct 29 21:57:20.176137 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:20.176070    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podStartSLOduration=1.176048163 podStartE2EDuration="1.176048163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:20.153847541 +0000 UTC m=+661.226804507" watchObservedRunningTime="2026-10-29 21:57:20.176048163 +0000 UTC m=+661.249005128"
Oct 29 21:57:20.401061 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:20.400863    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:57:29.400701 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:29.400669    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:57:29.412950 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:29.412928    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:57:30.205100 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:30.205065    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:57:39.725036 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:57:39.724831    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:57:53.694320 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.694270    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"]
Oct 29 21:57:53.698495 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.698473    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.713385 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.713367    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8a9e5f-openshift-default-dockercfg-98j45\""
Oct 29 21:57:53.727907 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.727882    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132ed775-c9e6-41f1-b224-fff550c6e073-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.728020 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.727931    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132ed775-c9e6-41f1-b224-fff550c6e073-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.728080 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.728045    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.728137 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.728102    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xdgm\" (UniqueName: \"kubernetes.io/projected/132ed775-c9e6-41f1-b224-fff550c6e073-kube-api-access-6xdgm\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.728212 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.728193    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132ed775-c9e6-41f1-b224-fff550c6e073-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.728268 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.728233    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.728362 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.728350    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.728431 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.728386    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.728431 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.728424    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.729327 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.729303    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"]
Oct 29 21:57:53.829332 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.829295    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132ed775-c9e6-41f1-b224-fff550c6e073-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.829332 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.829333    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132ed775-c9e6-41f1-b224-fff550c6e073-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.829572 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.829356    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.829572 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.829385    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xdgm\" (UniqueName: \"kubernetes.io/projected/132ed775-c9e6-41f1-b224-fff550c6e073-kube-api-access-6xdgm\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.829572 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.829448    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132ed775-c9e6-41f1-b224-fff550c6e073-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.829572 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.829479    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.829572 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.829558    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.829843 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.829591    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.829843 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.829770    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.829956 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.829866    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.829956 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.829935    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.830068 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.829967    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.830175 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.830149    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.830415 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.830398    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132ed775-c9e6-41f1-b224-fff550c6e073-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.831848 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.831832    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.832265 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.832242    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132ed775-c9e6-41f1-b224-fff550c6e073-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.853698 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.853673    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132ed775-c9e6-41f1-b224-fff550c6e073-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:53.853821 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:53.853805    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xdgm\" (UniqueName: \"kubernetes.io/projected/132ed775-c9e6-41f1-b224-fff550c6e073-kube-api-access-6xdgm\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:54.013157 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:54.013099    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:54.337794 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:54.337768    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"]
Oct 29 21:57:54.339192 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:57:54.339163    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod132ed775_c9e6_41f1_b224_fff550c6e073.slice/crio-89cbff96b976d2351e3a899b258d67827b48d64c002a283986e3a7b9992e3169 WatchSource:0}: Error finding container 89cbff96b976d2351e3a899b258d67827b48d64c002a283986e3a7b9992e3169: Status 404 returned error can't find the container with id 89cbff96b976d2351e3a899b258d67827b48d64c002a283986e3a7b9992e3169
Oct 29 21:57:54.341631 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:54.341593    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:57:54.341741 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:54.341670    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:57:54.341741 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:54.341704    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:57:55.344630 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:55.344590    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z" event={"ID":"132ed775-c9e6-41f1-b224-fff550c6e073","Type":"ContainerStarted","Data":"4fb526ee75ea34c842db621e76967cacd7903f6db2237b901b7372b19a60661b"}
Oct 29 21:57:55.345043 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:55.344650    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z" event={"ID":"132ed775-c9e6-41f1-b224-fff550c6e073","Type":"ContainerStarted","Data":"89cbff96b976d2351e3a899b258d67827b48d64c002a283986e3a7b9992e3169"}
Oct 29 21:57:56.013764 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:56.013729    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:57:56.015680 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:56.015654    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 29 21:57:56.015812 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:56.015703    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z" podUID="132ed775-c9e6-41f1-b224-fff550c6e073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 29 21:57:57.014168 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:57.014135    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 29 21:57:57.014584 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:57.014189    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z" podUID="132ed775-c9e6-41f1-b224-fff550c6e073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 29 21:57:58.014232 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:58.014201    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 29 21:57:58.014641 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:58.014260    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z" podUID="132ed775-c9e6-41f1-b224-fff550c6e073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 29 21:57:59.014119 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:59.014082    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 29 21:57:59.014335 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:57:59.014128    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z" podUID="132ed775-c9e6-41f1-b224-fff550c6e073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 29 21:58:00.013892 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:00.013849    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Oct 29 21:58:00.014061 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:00.013920    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z" podUID="132ed775-c9e6-41f1-b224-fff550c6e073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Oct 29 21:58:01.212911 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.212868    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:58:01.213775 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.213752    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:58:01.214986 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.214958    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 21:58:01.281681 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.281638    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z" podStartSLOduration=8.281626859 podStartE2EDuration="8.281626859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:57:55.412553091 +0000 UTC m=+696.485510055" watchObservedRunningTime="2026-10-29 21:58:01.281626859 +0000 UTC m=+702.354583822"
Oct 29 21:58:01.593083 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.593000    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"]
Oct 29 21:58:01.598389 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.598366    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.622575 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.622549    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"]
Oct 29 21:58:01.744553 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.744523    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.744706 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.744572    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.744706 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.744629    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.744706 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.744654    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.744818 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.744719    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.744818 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.744792    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htxvl\" (UniqueName: \"kubernetes.io/projected/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-kube-api-access-htxvl\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.744899 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.744831    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.744899 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.744847    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.744990 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.744913    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.846612 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.846547    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.846612 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.846588    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.846612 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.846611    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.846829 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.846626    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.846829 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.846691    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.846829 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.846741    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htxvl\" (UniqueName: \"kubernetes.io/projected/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-kube-api-access-htxvl\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.846829 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.846798    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.846829 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.846822    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.847077 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.846850    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.847077 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.846953    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.847077 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.846993    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.847224 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.847115    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.847224 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.847189    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.847671 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.847648    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.848714 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.848697    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.849337 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.849301    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.869741 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.869715    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.870119 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.870102    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htxvl\" (UniqueName: \"kubernetes.io/projected/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-kube-api-access-htxvl\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:01.912348 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:01.912326    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:02.232518 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:02.232493    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"]
Oct 29 21:58:02.233869 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:58:02.233843    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9762d8d0_a759_42ac_bc1b_c948adfa6f9d.slice/crio-f28c29c26c0cf41d80ea682dee47914e761161e5b6844025ee3363d91753dc35 WatchSource:0}: Error finding container f28c29c26c0cf41d80ea682dee47914e761161e5b6844025ee3363d91753dc35: Status 404 returned error can't find the container with id f28c29c26c0cf41d80ea682dee47914e761161e5b6844025ee3363d91753dc35
Oct 29 21:58:02.236759 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:02.236728    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:58:02.236841 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:02.236791    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:58:02.236841 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:02.236819    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:58:02.435017 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:02.434171    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5" event={"ID":"9762d8d0-a759-42ac-bc1b-c948adfa6f9d","Type":"ContainerStarted","Data":"f766610cabea37b19826f94ee2348ea64754f505943d291f6644290f47b70486"}
Oct 29 21:58:02.437126 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:02.437089    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5" event={"ID":"9762d8d0-a759-42ac-bc1b-c948adfa6f9d","Type":"ContainerStarted","Data":"f28c29c26c0cf41d80ea682dee47914e761161e5b6844025ee3363d91753dc35"}
Oct 29 21:58:02.480849 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:02.480754    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5" podStartSLOduration=1.480740501 podStartE2EDuration="1.480740501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:02.47883421 +0000 UTC m=+703.551791174" watchObservedRunningTime="2026-10-29 21:58:02.480740501 +0000 UTC m=+703.553697465"
Oct 29 21:58:02.912534 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:02.912490    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:03.927311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:03.923458    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 29 21:58:03.927311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:03.923527    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5" podUID="9762d8d0-a759-42ac-bc1b-c948adfa6f9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 29 21:58:04.923684 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:04.923644    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 29 21:58:04.923964 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:04.923935    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5" podUID="9762d8d0-a759-42ac-bc1b-c948adfa6f9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 29 21:58:05.917308 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:05.916462    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 29 21:58:05.917308 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:05.916525    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5" podUID="9762d8d0-a759-42ac-bc1b-c948adfa6f9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 29 21:58:06.914154 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:06.913671    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 29 21:58:06.914154 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:06.913777    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5" podUID="9762d8d0-a759-42ac-bc1b-c948adfa6f9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 29 21:58:07.912853 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:07.912816    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 29 21:58:07.913249 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:07.912874    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5" podUID="9762d8d0-a759-42ac-bc1b-c948adfa6f9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 29 21:58:08.912864 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:08.912822    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body=
Oct 29 21:58:08.913264 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:08.912893    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5" podUID="9762d8d0-a759-42ac-bc1b-c948adfa6f9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused"
Oct 29 21:58:09.720727 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:58:09.720693    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:58:10.482467 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:10.482433    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:10.482938 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:10.482755    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:10.483642 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:10.483620    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 21:58:18.714429 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.714394    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"]
Oct 29 21:58:18.719669 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.719641    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:58:18.747594 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.747565    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e4e24e\"/\"kube-root-ca.crt\""
Oct 29 21:58:18.747757 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.747609    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e4e24e\"/\"openshift-service-ca.crt\""
Oct 29 21:58:18.747832 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.747755    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e4e24e\"/\"default-dockercfg-8bczv\""
Oct 29 21:58:18.747891 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.747859    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e4e24e\"/\"e2e-conflict-a-eab27cf7-kserve-self-signed-certs\""
Oct 29 21:58:18.753732 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.753702    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"]
Oct 29 21:58:18.756845 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.756818    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-dshm\") pod \"e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q\" (UID: \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:58:18.756963 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.756881    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-tls-certs\") pod \"e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q\" (UID: \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:58:18.757544 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.757010    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-tmp-dir\") pod \"e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q\" (UID: \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:58:18.757544 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.757043    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-home\") pod \"e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q\" (UID: \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:58:18.757544 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.757138    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82h5t\" (UniqueName: \"kubernetes.io/projected/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-kube-api-access-82h5t\") pod \"e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q\" (UID: \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:58:18.757544 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.757228    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-model-cache\") pod \"e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q\" (UID: \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:58:18.858138 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.858109    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-dshm\") pod \"e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q\" (UID: \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:58:18.858330 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.858151    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-tls-certs\") pod \"e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q\" (UID: \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:58:18.858330 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.858198    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-tmp-dir\") pod \"e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q\" (UID: \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:58:18.858330 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.858217    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-home\") pod \"e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q\" (UID: \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:58:18.858330 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.858252    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82h5t\" (UniqueName: \"kubernetes.io/projected/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-kube-api-access-82h5t\") pod \"e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q\" (UID: \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:58:18.858540 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.858333    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-model-cache\") pod \"e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q\" (UID: \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:58:18.858676 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.858637    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-home\") pod \"e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q\" (UID: \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:58:18.858676 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.858666    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-tmp-dir\") pod \"e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q\" (UID: \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:58:18.858812 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.858750    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-model-cache\") pod \"e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q\" (UID: \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:58:18.860390 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.860374    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-dshm\") pod \"e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q\" (UID: \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:58:18.860803 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.860784    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-tls-certs\") pod \"e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q\" (UID: \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:58:18.881049 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:18.881025    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82h5t\" (UniqueName: \"kubernetes.io/projected/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-kube-api-access-82h5t\") pod \"e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q\" (UID: \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\") " pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:58:19.050297 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:19.050201    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e4e24e\"/\"default-dockercfg-8bczv\""
Oct 29 21:58:19.057527 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:19.057503    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:58:19.364998 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:19.364959    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"]
Oct 29 21:58:19.366558 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:58:19.366531    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58be6b18_3bda_47b5_8c32_42e0b2e87fc1.slice/crio-ba595625a6884578e5f3e2c0d0bec6f681666577b905f122b8c13f130bf68ce3 WatchSource:0}: Error finding container ba595625a6884578e5f3e2c0d0bec6f681666577b905f122b8c13f130bf68ce3: Status 404 returned error can't find the container with id ba595625a6884578e5f3e2c0d0bec6f681666577b905f122b8c13f130bf68ce3
Oct 29 21:58:19.572368 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:19.572329    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q" event={"ID":"58be6b18-3bda-47b5-8c32-42e0b2e87fc1","Type":"ContainerStarted","Data":"d21e243c987fcc0d35e494bff9bc3d6e0c7032e69def21552db363500c0244aa"}
Oct 29 21:58:19.572513 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:19.572385    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q" event={"ID":"58be6b18-3bda-47b5-8c32-42e0b2e87fc1","Type":"ContainerStarted","Data":"ba595625a6884578e5f3e2c0d0bec6f681666577b905f122b8c13f130bf68ce3"}
Oct 29 21:58:19.615056 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:19.614979    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q" podStartSLOduration=1.6149664289999999 podStartE2EDuration="1.614966429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:19.613405304 +0000 UTC m=+720.686362281" watchObservedRunningTime="2026-10-29 21:58:19.614966429 +0000 UTC m=+720.687923423"
Oct 29 21:58:20.058485 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:20.058454    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:58:21.250529 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:21.250496    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"]
Oct 29 21:58:21.254822 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:21.254796    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:58:21.270618 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:21.270592    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"]
Oct 29 21:58:21.279432 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:21.279410    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b31f3f\"/\"distinct-w4-b31f3f-kserve-self-signed-certs\""
Oct 29 21:58:21.301382 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:21.301357    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b51ab222-55f7-4fa1-b39d-0c48a06a7173-tls-certs\") pod \"distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c\" (UID: \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:58:21.301496 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:21.301459    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-dshm\") pod \"distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c\" (UID: \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:58:21.301554 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:21.301511    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-model-cache\") pod \"distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c\" (UID: \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:58:21.301612 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:21.301583    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-home\") pod \"distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c\" (UID: \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:58:21.301679 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:21.301643    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc8ts\" (UniqueName: \"kubernetes.io/projected/b51ab222-55f7-4fa1-b39d-0c48a06a7173-kube-api-access-fc8ts\") pod \"distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c\" (UID: \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:58:21.301679 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:21.301672    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-tmp-dir\") pod \"distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c\" (UID: \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:58:21.402137 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:21.402114    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-home\") pod \"distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c\" (UID: \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:58:21.402242 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:21.402161    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc8ts\" (UniqueName: \"kubernetes.io/projected/b51ab222-55f7-4fa1-b39d-0c48a06a7173-kube-api-access-fc8ts\") pod \"distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c\" (UID: \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:58:21.402242 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:21.402180    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-tmp-dir\") pod \"distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c\" (UID: \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:58:21.402242 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:21.402225    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b51ab222-55f7-4fa1-b39d-0c48a06a7173-tls-certs\") pod \"distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c\" (UID: \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:58:21.402391 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:21.402371    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-dshm\") pod \"distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c\" (UID: \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:58:21.402458 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:21.402439    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-model-cache\") pod \"distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c\" (UID: \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:58:21.402626 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:21.402605    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-home\") pod \"distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c\" (UID: \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:58:21.402702 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:21.402640    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-tmp-dir\") pod \"distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c\" (UID: \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:58:21.402777 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:21.402759    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-model-cache\") pod \"distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c\" (UID: \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:58:21.404658 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:21.404629    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-dshm\") pod \"distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c\" (UID: \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:58:21.405091 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:21.405076    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b51ab222-55f7-4fa1-b39d-0c48a06a7173-tls-certs\") pod \"distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c\" (UID: \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:58:21.425445 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:21.425423    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc8ts\" (UniqueName: \"kubernetes.io/projected/b51ab222-55f7-4fa1-b39d-0c48a06a7173-kube-api-access-fc8ts\") pod \"distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c\" (UID: \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:58:21.567635 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:21.567537    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:58:21.920068 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:21.920040    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"]
Oct 29 21:58:21.921069 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:58:21.921044    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb51ab222_55f7_4fa1_b39d_0c48a06a7173.slice/crio-df6e4611e40c409ebc5800fc2032a61ac9a66d86714b0b53f7e6081156280492 WatchSource:0}: Error finding container df6e4611e40c409ebc5800fc2032a61ac9a66d86714b0b53f7e6081156280492: Status 404 returned error can't find the container with id df6e4611e40c409ebc5800fc2032a61ac9a66d86714b0b53f7e6081156280492
Oct 29 21:58:22.616756 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:22.616720    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" event={"ID":"b51ab222-55f7-4fa1-b39d-0c48a06a7173","Type":"ContainerStarted","Data":"a77d84c5e175f77734846f3530da23ce5f1c6b6f6c657158598344e048a8a84c"}
Oct 29 21:58:22.617129 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:22.616768    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" event={"ID":"b51ab222-55f7-4fa1-b39d-0c48a06a7173","Type":"ContainerStarted","Data":"df6e4611e40c409ebc5800fc2032a61ac9a66d86714b0b53f7e6081156280492"}
Oct 29 21:58:22.663600 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:22.663551    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podStartSLOduration=1.663537528 podStartE2EDuration="1.663537528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:22.661207857 +0000 UTC m=+723.734164849" watchObservedRunningTime="2026-10-29 21:58:22.663537528 +0000 UTC m=+723.736494494"
Oct 29 21:58:23.568545 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:23.568503    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:58:29.057688 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:29.057652    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:58:29.070029 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:29.070006    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:58:29.705345 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:29.705304    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:58:31.571310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.569395    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:58:31.597365 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.597333    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:58:31.707537 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.707506    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"]
Oct 29 21:58:31.711757 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.711735    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:58:31.711875 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.711810    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.738834 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.738810    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"]
Oct 29 21:58:31.831797 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.831703    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.831797 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.831747    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6631eff-5d71-483c-aac7-8d578274ce76-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.831797 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.831768    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6631eff-5d71-483c-aac7-8d578274ce76-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.831797 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.831794    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6631eff-5d71-483c-aac7-8d578274ce76-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.832073 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.831909    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.832073 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.832037    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q88b5\" (UniqueName: \"kubernetes.io/projected/f6631eff-5d71-483c-aac7-8d578274ce76-kube-api-access-q88b5\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.832165 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.832136    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.832265 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.832204    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.832265 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.832249    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.939032 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.938990    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.941337 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.941303    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.941945 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.941920    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.942310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.942280    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.942462 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.942438    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6631eff-5d71-483c-aac7-8d578274ce76-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.942595 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.942577    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6631eff-5d71-483c-aac7-8d578274ce76-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.942711 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.942694    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6631eff-5d71-483c-aac7-8d578274ce76-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.942846 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.942826    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.943463 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.943433    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q88b5\" (UniqueName: \"kubernetes.io/projected/f6631eff-5d71-483c-aac7-8d578274ce76-kube-api-access-q88b5\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.944790 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.944768    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6631eff-5d71-483c-aac7-8d578274ce76-istiod-ca-cert\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.948320 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.946009    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-credential-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.948320 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.946271    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-workload-certs\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.948320 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.941119    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-workload-socket\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.948320 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.941835    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-istio-data\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.951397 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.951377    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-istio-envoy\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:31.951770 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:31.951747    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6631eff-5d71-483c-aac7-8d578274ce76-istio-podinfo\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:32.007943 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:32.001965    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6631eff-5d71-483c-aac7-8d578274ce76-istio-token\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:32.007943 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:32.002599    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q88b5\" (UniqueName: \"kubernetes.io/projected/f6631eff-5d71-483c-aac7-8d578274ce76-kube-api-access-q88b5\") pod \"e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") " pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:32.035596 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:32.035552    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:32.432830 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:32.432770    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"]
Oct 29 21:58:32.463739 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:32.463695    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:58:32.464345 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:32.464321    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:58:32.464522 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:32.464502    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:58:32.738362 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:32.737830    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw" event={"ID":"f6631eff-5d71-483c-aac7-8d578274ce76","Type":"ContainerStarted","Data":"31e08276344e9ca4d0dbd2b1b01de955fb414adce60b060b1b11350b0d6b9e7c"}
Oct 29 21:58:32.738362 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:32.737890    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw" event={"ID":"f6631eff-5d71-483c-aac7-8d578274ce76","Type":"ContainerStarted","Data":"e6184dedc5ce30abdbfd447ace2eb9b1852cd90f3452136f89c20f1171685909"}
Oct 29 21:58:32.797415 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:32.792265    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw" podStartSLOduration=1.79224549 podStartE2EDuration="1.79224549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:32.790801544 +0000 UTC m=+733.863758518" watchObservedRunningTime="2026-10-29 21:58:32.79224549 +0000 UTC m=+733.865202455"
Oct 29 21:58:33.046361 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:33.043370    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:33.046361 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:33.043529    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 29 21:58:33.046361 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:33.043576    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw" podUID="f6631eff-5d71-483c-aac7-8d578274ce76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 29 21:58:34.043312 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:34.042705    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 29 21:58:34.043312 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:34.042764    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw" podUID="f6631eff-5d71-483c-aac7-8d578274ce76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 29 21:58:35.040618 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:35.040580    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 29 21:58:35.040798 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:35.040648    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw" podUID="f6631eff-5d71-483c-aac7-8d578274ce76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 29 21:58:36.040491 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:36.040455    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 29 21:58:36.040891 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:36.040522    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw" podUID="f6631eff-5d71-483c-aac7-8d578274ce76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 29 21:58:37.040559 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:37.040522    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 29 21:58:37.040952 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:37.040583    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw" podUID="f6631eff-5d71-483c-aac7-8d578274ce76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 29 21:58:38.040927 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:38.040882    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 29 21:58:38.041439 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:38.040963    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw" podUID="f6631eff-5d71-483c-aac7-8d578274ce76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 29 21:58:39.040234 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:39.040202    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 29 21:58:39.040424 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:39.040252    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw" podUID="f6631eff-5d71-483c-aac7-8d578274ce76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 29 21:58:39.722632 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:58:39.722589    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:58:40.040238 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:40.040160    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Oct 29 21:58:40.040238 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:40.040221    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw" podUID="f6631eff-5d71-483c-aac7-8d578274ce76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Oct 29 21:58:41.044510 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:41.044480    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:41.044946 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:41.044723    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:41.045407 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:41.045386    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 21:58:50.489277 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:50.489247    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"]
Oct 29 21:58:50.493896 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:50.493874    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:58:50.519833 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:50.519809    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8a9e5f\"/\"distinct-w3-8a9e5f-kserve-self-signed-certs\""
Oct 29 21:58:50.522709 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:50.522685    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"]
Oct 29 21:58:50.627465 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:50.627429    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-model-cache\") pod \"distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t\" (UID: \"b9303c60-e8b1-4f7c-841f-811099ba8660\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:58:50.627610 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:50.627508    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-home\") pod \"distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t\" (UID: \"b9303c60-e8b1-4f7c-841f-811099ba8660\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:58:50.627653 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:50.627633    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9303c60-e8b1-4f7c-841f-811099ba8660-tls-certs\") pod \"distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t\" (UID: \"b9303c60-e8b1-4f7c-841f-811099ba8660\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:58:50.627699 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:50.627687    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-dshm\") pod \"distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t\" (UID: \"b9303c60-e8b1-4f7c-841f-811099ba8660\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:58:50.627801 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:50.627769    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c4ck\" (UniqueName: \"kubernetes.io/projected/b9303c60-e8b1-4f7c-841f-811099ba8660-kube-api-access-9c4ck\") pod \"distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t\" (UID: \"b9303c60-e8b1-4f7c-841f-811099ba8660\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:58:50.627851 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:50.627840    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-tmp-dir\") pod \"distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t\" (UID: \"b9303c60-e8b1-4f7c-841f-811099ba8660\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:58:50.736097 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:50.736052    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-dshm\") pod \"distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t\" (UID: \"b9303c60-e8b1-4f7c-841f-811099ba8660\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:58:50.738400 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:50.738368    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c4ck\" (UniqueName: \"kubernetes.io/projected/b9303c60-e8b1-4f7c-841f-811099ba8660-kube-api-access-9c4ck\") pod \"distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t\" (UID: \"b9303c60-e8b1-4f7c-841f-811099ba8660\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:58:50.738722 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:50.738695    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-tmp-dir\") pod \"distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t\" (UID: \"b9303c60-e8b1-4f7c-841f-811099ba8660\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:58:50.738914 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:50.738899    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-model-cache\") pod \"distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t\" (UID: \"b9303c60-e8b1-4f7c-841f-811099ba8660\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:58:50.739043 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:50.739030    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-home\") pod \"distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t\" (UID: \"b9303c60-e8b1-4f7c-841f-811099ba8660\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:58:50.739244 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:50.739225    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9303c60-e8b1-4f7c-841f-811099ba8660-tls-certs\") pod \"distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t\" (UID: \"b9303c60-e8b1-4f7c-841f-811099ba8660\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:58:50.758383 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:50.740199    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-tmp-dir\") pod \"distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t\" (UID: \"b9303c60-e8b1-4f7c-841f-811099ba8660\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:58:50.758641 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:50.741087    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-model-cache\") pod \"distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t\" (UID: \"b9303c60-e8b1-4f7c-841f-811099ba8660\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:58:50.758749 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:50.741737    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-home\") pod \"distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t\" (UID: \"b9303c60-e8b1-4f7c-841f-811099ba8660\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:58:50.758833 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:50.755626    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-dshm\") pod \"distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t\" (UID: \"b9303c60-e8b1-4f7c-841f-811099ba8660\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:58:50.759613 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:50.759586    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9303c60-e8b1-4f7c-841f-811099ba8660-tls-certs\") pod \"distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t\" (UID: \"b9303c60-e8b1-4f7c-841f-811099ba8660\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:58:50.775862 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:50.775826    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c4ck\" (UniqueName: \"kubernetes.io/projected/b9303c60-e8b1-4f7c-841f-811099ba8660-kube-api-access-9c4ck\") pod \"distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t\" (UID: \"b9303c60-e8b1-4f7c-841f-811099ba8660\") " pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:58:50.807634 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:50.807604    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:58:51.116156 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:51.116130    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"]
Oct 29 21:58:51.118120 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:58:51.118092    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9303c60_e8b1_4f7c_841f_811099ba8660.slice/crio-7a2f4f22fd9641caa081aec3a6d4f798d28e7411f1fc5dd55a07232a05e8ea37 WatchSource:0}: Error finding container 7a2f4f22fd9641caa081aec3a6d4f798d28e7411f1fc5dd55a07232a05e8ea37: Status 404 returned error can't find the container with id 7a2f4f22fd9641caa081aec3a6d4f798d28e7411f1fc5dd55a07232a05e8ea37
Oct 29 21:58:51.924809 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:51.924774    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" event={"ID":"b9303c60-e8b1-4f7c-841f-811099ba8660","Type":"ContainerStarted","Data":"780fb9a873de22c245f9980b855ad6f272ca0f63bc80a67e728adfdecdb420e3"}
Oct 29 21:58:51.925198 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:51.924819    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" event={"ID":"b9303c60-e8b1-4f7c-841f-811099ba8660","Type":"ContainerStarted","Data":"7a2f4f22fd9641caa081aec3a6d4f798d28e7411f1fc5dd55a07232a05e8ea37"}
Oct 29 21:58:51.968023 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:51.967973    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podStartSLOduration=1.967959953 podStartE2EDuration="1.967959953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:51.966927645 +0000 UTC m=+753.039884609" watchObservedRunningTime="2026-10-29 21:58:51.967959953 +0000 UTC m=+753.040916916"
Oct 29 21:58:52.808366 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:52.808335    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:58:53.896835 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:53.896802    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"]
Oct 29 21:58:53.900777 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:53.900760    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:53.947570 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:53.947542    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"]
Oct 29 21:58:53.982278 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:53.982247    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:53.982440 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:53.982280    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:53.982440 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:53.982375    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91917d2-1f66-4ee4-a312-57398651739e-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:53.982440 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:53.982422    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91917d2-1f66-4ee4-a312-57398651739e-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:53.982629 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:53.982613    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:53.982663 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:53.982644    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:53.982693 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:53.982678    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91917d2-1f66-4ee4-a312-57398651739e-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:53.982800 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:53.982786    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzqfj\" (UniqueName: \"kubernetes.io/projected/d91917d2-1f66-4ee4-a312-57398651739e-kube-api-access-vzqfj\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:53.982838 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:53.982825    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:54.084078 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.084044    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:54.084078 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.084080    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:54.084250 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.084102    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91917d2-1f66-4ee4-a312-57398651739e-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:54.084250 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.084165    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzqfj\" (UniqueName: \"kubernetes.io/projected/d91917d2-1f66-4ee4-a312-57398651739e-kube-api-access-vzqfj\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:54.084250 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.084194    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:54.084250 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.084215    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:54.084250 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.084229    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:54.084250 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.084249    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91917d2-1f66-4ee4-a312-57398651739e-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:54.084569 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.084305    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91917d2-1f66-4ee4-a312-57398651739e-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:54.084627 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.084601    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:54.084695 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.084670    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:54.084800 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.084776    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:54.084867 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.084855    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:54.084992 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.084920    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91917d2-1f66-4ee4-a312-57398651739e-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:54.086368 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.086346    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:54.086955 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.086933    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91917d2-1f66-4ee4-a312-57398651739e-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:54.107523 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.107500    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzqfj\" (UniqueName: \"kubernetes.io/projected/d91917d2-1f66-4ee4-a312-57398651739e-kube-api-access-vzqfj\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:54.107613 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.107536    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91917d2-1f66-4ee4-a312-57398651739e-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:54.217450 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.217425    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:54.539747 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.539720    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"]
Oct 29 21:58:54.541029 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:58:54.541004    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91917d2_1f66_4ee4_a312_57398651739e.slice/crio-f680282948b430472f66ef341e19df3efca6c5e30a97a1ea9a815516120a5443 WatchSource:0}: Error finding container f680282948b430472f66ef341e19df3efca6c5e30a97a1ea9a815516120a5443: Status 404 returned error can't find the container with id f680282948b430472f66ef341e19df3efca6c5e30a97a1ea9a815516120a5443
Oct 29 21:58:54.543458 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.543427    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:58:54.543534 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.543495    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:58:54.543534 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.543524    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:58:54.956829 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.956799    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr" event={"ID":"d91917d2-1f66-4ee4-a312-57398651739e","Type":"ContainerStarted","Data":"bc30e57de07692cb4a3806e7b3e8b9681bbed19d93e483b2fc26d80bd7f5d165"}
Oct 29 21:58:54.957246 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:54.956846    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr" event={"ID":"d91917d2-1f66-4ee4-a312-57398651739e","Type":"ContainerStarted","Data":"f680282948b430472f66ef341e19df3efca6c5e30a97a1ea9a815516120a5443"}
Oct 29 21:58:55.005615 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:55.005570    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr" podStartSLOduration=2.005556214 podStartE2EDuration="2.005556214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:58:55.002689126 +0000 UTC m=+756.075646091" watchObservedRunningTime="2026-10-29 21:58:55.005556214 +0000 UTC m=+756.078513226"
Oct 29 21:58:55.218564 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:55.218473    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:58:55.219929 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:55.219908    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 29 21:58:55.219992 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:55.219966    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr" podUID="d91917d2-1f66-4ee4-a312-57398651739e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 29 21:58:56.218101 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:56.218060    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 29 21:58:56.218530 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:56.218115    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr" podUID="d91917d2-1f66-4ee4-a312-57398651739e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 29 21:58:57.218104 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:57.218070    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 29 21:58:57.218519 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:57.218121    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr" podUID="d91917d2-1f66-4ee4-a312-57398651739e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 29 21:58:58.218031 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:58.217995    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 29 21:58:58.218215 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:58.218056    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr" podUID="d91917d2-1f66-4ee4-a312-57398651739e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 29 21:58:59.236310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:59.229091    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 29 21:58:59.236310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:59.229174    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr" podUID="d91917d2-1f66-4ee4-a312-57398651739e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 29 21:58:59.791490 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:59.791432    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:58:59.823422 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:59.823368    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 21:58:59.931434 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:58:59.931380    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:00.110472 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.110372    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q" podUID="58be6b18-3bda-47b5-8c32-42e0b2e87fc1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:00.126310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.113154    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"]
Oct 29 21:59:00.133964 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.133934    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.249311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.221209    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.249311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.221259    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.249311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.221470    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.249311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.221501    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tv56\" (UniqueName: \"kubernetes.io/projected/16540bea-446b-46d9-9cde-4c9c34785ce8-kube-api-access-6tv56\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.249311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.221649    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.249311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.221710    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.249311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.221829    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.249311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.221873    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.249311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.221903    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16540bea-446b-46d9-9cde-4c9c34785ce8-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.249311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.245377    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:00.354510 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.296628    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"]
Oct 29 21:59:00.354510 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.318065    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 29 21:59:00.354510 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.318122    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr" podUID="d91917d2-1f66-4ee4-a312-57398651739e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 29 21:59:00.354510 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.340172    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.354510 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.340217    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.354510 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.340246    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16540bea-446b-46d9-9cde-4c9c34785ce8-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.354510 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.340320    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.354510 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.340345    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.354510 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.340445    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.354510 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.340472    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tv56\" (UniqueName: \"kubernetes.io/projected/16540bea-446b-46d9-9cde-4c9c34785ce8-kube-api-access-6tv56\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.354510 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.340601    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.354510 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.340657    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.354510 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.341214    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.354510 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.341510    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.354510 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.342476    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16540bea-446b-46d9-9cde-4c9c34785ce8-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.412358 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.405789    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.412358 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.406215    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.505309 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.485137    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.558311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.555487    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.558311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.558209    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tv56\" (UniqueName: \"kubernetes.io/projected/16540bea-446b-46d9-9cde-4c9c34785ce8-kube-api-access-6tv56\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.563314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.562920    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.654309 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.653678    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:00.707308 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.697373    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q" podUID="58be6b18-3bda-47b5-8c32-42e0b2e87fc1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:00.711304 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.710427    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:00.792338 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.792272    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:00.792939 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.792917    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:59:00.811438 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.810442    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:59:00.828029 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.827988    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:00.858309 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.828661    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:59:01.004889 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.961749    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:01.004889 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:00.962238    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:59:01.268360 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:01.259459    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:01.268360 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:01.262796    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 29 21:59:01.309322 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:01.304650    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr" podUID="d91917d2-1f66-4ee4-a312-57398651739e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 29 21:59:01.368319 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:01.337189    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:59:01.798769 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:01.735414    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 21:59:01.798769 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:01.736086    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:59:01.798769 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:01.795403    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:01.798769 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:01.739370    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q" podUID="58be6b18-3bda-47b5-8c32-42e0b2e87fc1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 29 21:59:01.835030 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:01.812503    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:59:01.835030 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:01.832726    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 21:59:01.835030 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:01.833662    4962 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:01.978389 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:01.962511    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:02.280360 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:02.237643    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 29 21:59:02.280360 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:02.237697    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr" podUID="d91917d2-1f66-4ee4-a312-57398651739e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 29 21:59:02.283053 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:02.283018    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:02.380731 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:02.380678    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:59:02.383436 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:02.383407    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:59:02.383674 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:02.383638    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:59:02.398580 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:02.398555    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"]
Oct 29 21:59:02.489652 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:02.469621    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 21:59:02.751645 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:02.735158    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded"
Oct 29 21:59:02.826409 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:02.826362    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:02.839426 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:02.839389    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q" podUID="58be6b18-3bda-47b5-8c32-42e0b2e87fc1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 29 21:59:02.845450 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:02.840355    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:02.895511 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:02.895484    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:59:02.949361 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:02.948608    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:59:03.015711 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:03.013631    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg" event={"ID":"16540bea-446b-46d9-9cde-4c9c34785ce8","Type":"ContainerStarted","Data":"4f2526809fa47c0d68df65f7bfcd32b72b0039c3f2c52a539103ce864273c176"}
Oct 29 21:59:03.067347 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:03.066730    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:59:03.100337 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:03.098175    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:59:03.107032 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:03.106402    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:59:03.218805 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:03.218772    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 29 21:59:03.218988 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:03.218843    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr" podUID="d91917d2-1f66-4ee4-a312-57398651739e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 29 21:59:04.018530 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:04.018494    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg" event={"ID":"16540bea-446b-46d9-9cde-4c9c34785ce8","Type":"ContainerStarted","Data":"62b9ff8dbf2dcecae744e4db1bfaeecc0c29f79de97fed607153dbf915f5ec18"}
Oct 29 21:59:04.089351 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:04.089275    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg" podStartSLOduration=4.089254812 podStartE2EDuration="4.089254812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:04.085195756 +0000 UTC m=+765.158152721" watchObservedRunningTime="2026-10-29 21:59:04.089254812 +0000 UTC m=+765.162211778"
Oct 29 21:59:04.654958 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:04.654926    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:04.656281 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:04.656254    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 29 21:59:04.656456 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:04.656340    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg" podUID="16540bea-446b-46d9-9cde-4c9c34785ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 29 21:59:05.110494 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.110404    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:59:05.110941 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.110785    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:59:05.111706 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.111684    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 21:59:05.157394 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.157368    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"]
Oct 29 21:59:05.161631 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.161610    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.222959 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.222921    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"]
Oct 29 21:59:05.270591 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.270522    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c5ec48-f48a-4703-a82c-9a6995506125-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.270908 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.270604    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.270908 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.270630    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.270908 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.270790    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c5ec48-f48a-4703-a82c-9a6995506125-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.271221 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.270916    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c5ec48-f48a-4703-a82c-9a6995506125-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.271221 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.270973    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbhq7\" (UniqueName: \"kubernetes.io/projected/10c5ec48-f48a-4703-a82c-9a6995506125-kube-api-access-lbhq7\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.300311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.283454    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.300311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.283551    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.300311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.283580    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.394360 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.392983    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbhq7\" (UniqueName: \"kubernetes.io/projected/10c5ec48-f48a-4703-a82c-9a6995506125-kube-api-access-lbhq7\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.394360 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.393082    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.394360 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.393163    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.394360 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.393189    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.394360 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.393270    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c5ec48-f48a-4703-a82c-9a6995506125-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.394360 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.393369    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.394360 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.393398    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.394360 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.393451    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c5ec48-f48a-4703-a82c-9a6995506125-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.394360 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.393527    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c5ec48-f48a-4703-a82c-9a6995506125-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.394966 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.394678    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.396715 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.396686    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.397101 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.397076    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.398277 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.397854    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.398484 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.398455    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c5ec48-f48a-4703-a82c-9a6995506125-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.399109 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.399079    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.399595 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.399570    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c5ec48-f48a-4703-a82c-9a6995506125-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.418099 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.418071    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbhq7\" (UniqueName: \"kubernetes.io/projected/10c5ec48-f48a-4703-a82c-9a6995506125-kube-api-access-lbhq7\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.419244 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.419221    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c5ec48-f48a-4703-a82c-9a6995506125-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.523357 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.518375    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:05.655206 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.655181    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 29 21:59:05.655372 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.655227    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg" podUID="16540bea-446b-46d9-9cde-4c9c34785ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 29 21:59:05.900023 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.899994    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"]
Oct 29 21:59:05.901615 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:59:05.901584    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c5ec48_f48a_4703_a82c_9a6995506125.slice/crio-261f778cd543b1e36e42073cd85349d02847a1a1a02f32b04aa2dfdbd6ec7f85 WatchSource:0}: Error finding container 261f778cd543b1e36e42073cd85349d02847a1a1a02f32b04aa2dfdbd6ec7f85: Status 404 returned error can't find the container with id 261f778cd543b1e36e42073cd85349d02847a1a1a02f32b04aa2dfdbd6ec7f85
Oct 29 21:59:05.904454 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.904425    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:59:05.904524 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.904491    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:59:05.904524 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:05.904519    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:59:06.034792 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:06.034746    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr" event={"ID":"10c5ec48-f48a-4703-a82c-9a6995506125","Type":"ContainerStarted","Data":"c668a05f6f3d9dbc6215843071b014aba97a0397eae32e6077fb52f4bda89af1"}
Oct 29 21:59:06.034966 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:06.034821    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr" event={"ID":"10c5ec48-f48a-4703-a82c-9a6995506125","Type":"ContainerStarted","Data":"261f778cd543b1e36e42073cd85349d02847a1a1a02f32b04aa2dfdbd6ec7f85"}
Oct 29 21:59:06.102652 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:06.102592    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr" podStartSLOduration=1.102573504 podStartE2EDuration="1.102573504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:06.099841294 +0000 UTC m=+767.172798277" watchObservedRunningTime="2026-10-29 21:59:06.102573504 +0000 UTC m=+767.175530471"
Oct 29 21:59:06.519518 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:06.519487    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:06.521352 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:06.521316    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 29 21:59:06.521475 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:06.521373    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr" podUID="10c5ec48-f48a-4703-a82c-9a6995506125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 29 21:59:06.655246 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:06.655214    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 29 21:59:06.655410 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:06.655274    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg" podUID="16540bea-446b-46d9-9cde-4c9c34785ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 29 21:59:07.521312 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:07.520682    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 29 21:59:07.521312 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:07.520749    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr" podUID="10c5ec48-f48a-4703-a82c-9a6995506125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 29 21:59:07.659315 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:07.657667    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 29 21:59:07.659315 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:07.657727    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg" podUID="16540bea-446b-46d9-9cde-4c9c34785ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 29 21:59:07.719311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:07.718866    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"]
Oct 29 21:59:07.750583 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:07.750543    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"]
Oct 29 21:59:07.750745 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:07.750642    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 21:59:07.768777 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:07.768090    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b31f3f\"/\"distinct-2-w4-b31f3f-kserve-self-signed-certs\""
Oct 29 21:59:07.842966 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:07.842864    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-model-cache\") pod \"distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p\" (UID: \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 21:59:07.843141 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:07.842977    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-home\") pod \"distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p\" (UID: \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 21:59:07.843141 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:07.843011    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-tmp-dir\") pod \"distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p\" (UID: \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 21:59:07.843141 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:07.843041    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwq2q\" (UniqueName: \"kubernetes.io/projected/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-kube-api-access-vwq2q\") pod \"distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p\" (UID: \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 21:59:07.843364 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:07.843246    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-tls-certs\") pod \"distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p\" (UID: \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 21:59:07.843364 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:07.843303    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-dshm\") pod \"distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p\" (UID: \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 21:59:07.947185 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:07.947127    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-home\") pod \"distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p\" (UID: \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 21:59:07.947946 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:07.947921    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-tmp-dir\") pod \"distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p\" (UID: \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 21:59:07.948466 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:07.948438    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwq2q\" (UniqueName: \"kubernetes.io/projected/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-kube-api-access-vwq2q\") pod \"distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p\" (UID: \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 21:59:07.949028 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:07.949007    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-tls-certs\") pod \"distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p\" (UID: \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 21:59:07.966200 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:07.966166    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-dshm\") pod \"distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p\" (UID: \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 21:59:07.966973 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:07.966948    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-model-cache\") pod \"distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p\" (UID: \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 21:59:07.967221 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:07.947835    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-home\") pod \"distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p\" (UID: \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 21:59:07.967364 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:07.948372    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-tmp-dir\") pod \"distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p\" (UID: \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 21:59:07.967766 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:07.967743    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-model-cache\") pod \"distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p\" (UID: \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 21:59:08.005315 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:07.998420    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-dshm\") pod \"distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p\" (UID: \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 21:59:08.005551 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:08.000171    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-tls-certs\") pod \"distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p\" (UID: \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 21:59:08.017164 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:08.017080    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwq2q\" (UniqueName: \"kubernetes.io/projected/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-kube-api-access-vwq2q\") pod \"distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p\" (UID: \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\") " pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 21:59:08.097712 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:08.097614    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 21:59:08.522418 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:08.521724    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 29 21:59:08.522418 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:08.521787    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr" podUID="10c5ec48-f48a-4703-a82c-9a6995506125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 29 21:59:08.623340 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:08.623108    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"]
Oct 29 21:59:08.627720 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:59:08.627678    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode548eb7e_ffd6_48b0_8358_30a081d3cf4b.slice/crio-7681049fcaa4b56fe30826133bb0f9ad2a71503198091eceaddeae13ce717579 WatchSource:0}: Error finding container 7681049fcaa4b56fe30826133bb0f9ad2a71503198091eceaddeae13ce717579: Status 404 returned error can't find the container with id 7681049fcaa4b56fe30826133bb0f9ad2a71503198091eceaddeae13ce717579
Oct 29 21:59:08.654744 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:08.654716    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 29 21:59:08.654833 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:08.654760    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg" podUID="16540bea-446b-46d9-9cde-4c9c34785ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 29 21:59:09.141096 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:09.141061    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" event={"ID":"e548eb7e-ffd6-48b0-8358-30a081d3cf4b","Type":"ContainerStarted","Data":"150138e807a2048dcc0dc7554b609bcd44307c79135a8e9467f8320806e82566"}
Oct 29 21:59:09.141302 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:09.141111    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" event={"ID":"e548eb7e-ffd6-48b0-8358-30a081d3cf4b","Type":"ContainerStarted","Data":"7681049fcaa4b56fe30826133bb0f9ad2a71503198091eceaddeae13ce717579"}
Oct 29 21:59:09.200313 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:09.193850    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podStartSLOduration=2.193831422 podStartE2EDuration="2.193831422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:09.192360455 +0000 UTC m=+770.265317444" watchObservedRunningTime="2026-10-29 21:59:09.193831422 +0000 UTC m=+770.266788389"
Oct 29 21:59:09.519257 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:09.519215    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 29 21:59:09.519447 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:09.519326    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr" podUID="10c5ec48-f48a-4703-a82c-9a6995506125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 29 21:59:09.654993 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:09.654962    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 29 21:59:09.655400 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:09.655011    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg" podUID="16540bea-446b-46d9-9cde-4c9c34785ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 29 21:59:09.728563 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:59:09.728340    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:59:09.805002 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:09.804887    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"]
Oct 29 21:59:09.809103 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:09.809085    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:09.877314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:09.876351    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"]
Oct 29 21:59:09.956692 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:09.956649    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:09.956924 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:09.956896    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:09.957039 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:09.957026    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:09.957172 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:09.957158    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:09.957342 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:09.957328    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:09.957642 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:09.957563    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:09.957836 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:09.957821    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsjfc\" (UniqueName: \"kubernetes.io/projected/35c2b463-e2a6-4b1d-8425-558ca7832668-kube-api-access-wsjfc\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:09.958012 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:09.957982    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c2b463-e2a6-4b1d-8425-558ca7832668-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:09.958084 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:09.958064    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:10.059526 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.059439    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:10.059526 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.059495    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsjfc\" (UniqueName: \"kubernetes.io/projected/35c2b463-e2a6-4b1d-8425-558ca7832668-kube-api-access-wsjfc\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:10.059730 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.059545    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c2b463-e2a6-4b1d-8425-558ca7832668-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:10.059730 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.059588    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:10.059730 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.059649    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:10.059730 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.059679    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:10.059730 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.059716    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:10.059978 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.059760    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:10.059978 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.059783    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:10.060093 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.060039    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:10.060093 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.060083    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:10.060187 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.060112    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:10.060251 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.060221    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:10.060729 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.060702    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c2b463-e2a6-4b1d-8425-558ca7832668-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:10.062092 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.062061    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:10.062390 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.062374    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:10.102400 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.102361    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 21:59:10.105300 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.105242    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsjfc\" (UniqueName: \"kubernetes.io/projected/35c2b463-e2a6-4b1d-8425-558ca7832668-kube-api-access-wsjfc\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:10.105787 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.105762    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:10.131199 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.131167    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:10.521413 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.520939    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 29 21:59:10.521413 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.520999    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr" podUID="10c5ec48-f48a-4703-a82c-9a6995506125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 29 21:59:10.586313 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.578809    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"]
Oct 29 21:59:10.628312 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:59:10.621570    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c2b463_e2a6_4b1d_8425_558ca7832668.slice/crio-6a803f26456110552424f7e4ef6beeaf8df1237237b58b4e034c66b702bf78ee WatchSource:0}: Error finding container 6a803f26456110552424f7e4ef6beeaf8df1237237b58b4e034c66b702bf78ee: Status 404 returned error can't find the container with id 6a803f26456110552424f7e4ef6beeaf8df1237237b58b4e034c66b702bf78ee
Oct 29 21:59:10.628312 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.623953    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:59:10.628312 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.624039    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:59:10.628312 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.624084    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:59:10.834349 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.833698    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:59:10.852032 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:10.851955    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:59:11.090301 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:11.088637    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:11.090301 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:11.090084    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:11.096517 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:11.096495    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 21:59:11.157872 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:11.157830    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc" event={"ID":"35c2b463-e2a6-4b1d-8425-558ca7832668","Type":"ContainerStarted","Data":"ee42b17744a6a1fc61efde1f2211e522f8707a0f8a5b4888967a0acae6e3ab33"}
Oct 29 21:59:11.158068 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:11.157908    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc" event={"ID":"35c2b463-e2a6-4b1d-8425-558ca7832668","Type":"ContainerStarted","Data":"6a803f26456110552424f7e4ef6beeaf8df1237237b58b4e034c66b702bf78ee"}
Oct 29 21:59:11.248897 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:11.248846    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc" podStartSLOduration=2.248832036 podStartE2EDuration="2.248832036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:11.246771922 +0000 UTC m=+772.319728907" watchObservedRunningTime="2026-10-29 21:59:11.248832036 +0000 UTC m=+772.321789002"
Oct 29 21:59:11.525018 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:11.524980    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 29 21:59:11.525234 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:11.525122    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr" podUID="10c5ec48-f48a-4703-a82c-9a6995506125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 29 21:59:12.133225 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:12.133051    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:12.133790 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:12.133760    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 29 21:59:12.133887 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:12.133816    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc" podUID="35c2b463-e2a6-4b1d-8425-558ca7832668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 29 21:59:12.519500 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:12.519461    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 29 21:59:12.519749 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:12.519725    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr" podUID="10c5ec48-f48a-4703-a82c-9a6995506125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 29 21:59:13.132452 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:13.132414    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 29 21:59:13.132654 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:13.132470    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc" podUID="35c2b463-e2a6-4b1d-8425-558ca7832668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 29 21:59:13.568982 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:13.568948    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:13.569624 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:13.569598    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:13.570761 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:13.570713    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 21:59:14.132440 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:14.132403    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 29 21:59:14.132617 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:14.132463    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc" podUID="35c2b463-e2a6-4b1d-8425-558ca7832668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 29 21:59:15.138171 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.138131    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 29 21:59:15.138792 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.138754    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc" podUID="35c2b463-e2a6-4b1d-8425-558ca7832668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 29 21:59:15.163752 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.163708    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"]
Oct 29 21:59:15.169137 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.169114    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.205658 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.205627    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"]
Oct 29 21:59:15.303396 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.303369    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhdvj\" (UniqueName: \"kubernetes.io/projected/f06b758b-d09f-4882-a198-d88e91c7f390-kube-api-access-rhdvj\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.303596 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.303416    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.303596 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.303563    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.303723 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.303699    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.303764 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.303730    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f06b758b-d09f-4882-a198-d88e91c7f390-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.303810 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.303768    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.303810 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.303790    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.303942 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.303920    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f06b758b-d09f-4882-a198-d88e91c7f390-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.303985 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.303964    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f06b758b-d09f-4882-a198-d88e91c7f390-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.404945 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.404902    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.405112 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.405030    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.405112 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.405059    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f06b758b-d09f-4882-a198-d88e91c7f390-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.405230 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.405111    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.405230 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.405151    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.405230 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.405207    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f06b758b-d09f-4882-a198-d88e91c7f390-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.405388 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.405242    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f06b758b-d09f-4882-a198-d88e91c7f390-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.405388 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.405375    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.405492 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.405408    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhdvj\" (UniqueName: \"kubernetes.io/projected/f06b758b-d09f-4882-a198-d88e91c7f390-kube-api-access-rhdvj\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.405492 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.405474    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.405492 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.405484    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.405652 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.405521    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.405756 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.405733    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.406011 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.405990    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f06b758b-d09f-4882-a198-d88e91c7f390-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.407721 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.407702    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.407835 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.407815    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f06b758b-d09f-4882-a198-d88e91c7f390-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.428388 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.428364    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f06b758b-d09f-4882-a198-d88e91c7f390-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.428513 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.428493    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhdvj\" (UniqueName: \"kubernetes.io/projected/f06b758b-d09f-4882-a198-d88e91c7f390-kube-api-access-rhdvj\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.519962 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.519935    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:15.869089 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.869058    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"]
Oct 29 21:59:15.870370 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:59:15.870343    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06b758b_d09f_4882_a198_d88e91c7f390.slice/crio-a6dbed014094c123f4b7fe0106d58deb9157fd004f814d465cf439fc4765ffc2 WatchSource:0}: Error finding container a6dbed014094c123f4b7fe0106d58deb9157fd004f814d465cf439fc4765ffc2: Status 404 returned error can't find the container with id a6dbed014094c123f4b7fe0106d58deb9157fd004f814d465cf439fc4765ffc2
Oct 29 21:59:15.897313 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.895259    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:59:15.897313 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.895377    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:59:15.897313 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:15.895425    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:59:16.132221 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:16.132140    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 29 21:59:16.132221 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:16.132188    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc" podUID="35c2b463-e2a6-4b1d-8425-558ca7832668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 29 21:59:16.222918 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:16.222882    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh" event={"ID":"f06b758b-d09f-4882-a198-d88e91c7f390","Type":"ContainerStarted","Data":"e98eaf67c7a73837aedc222b40171dc7f8248fe0383cc321e7fb6b3861bf36f0"}
Oct 29 21:59:16.223319 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:16.222942    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh" event={"ID":"f06b758b-d09f-4882-a198-d88e91c7f390","Type":"ContainerStarted","Data":"a6dbed014094c123f4b7fe0106d58deb9157fd004f814d465cf439fc4765ffc2"}
Oct 29 21:59:16.520122 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:16.520092    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:16.521620 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:16.521598    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 29 21:59:16.521690 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:16.521646    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh" podUID="f06b758b-d09f-4882-a198-d88e91c7f390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 29 21:59:17.131953 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:17.131914    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 29 21:59:17.132134 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:17.131978    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc" podUID="35c2b463-e2a6-4b1d-8425-558ca7832668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 29 21:59:17.521068 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:17.521029    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 29 21:59:17.521551 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:17.521106    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh" podUID="f06b758b-d09f-4882-a198-d88e91c7f390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 29 21:59:18.101988 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:18.101948    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 21:59:18.115406 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:18.115378    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 21:59:18.135540 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:18.135519    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:18.135797 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:18.135773    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:18.136480 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:18.136464    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 21:59:18.161429 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:18.161388    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh" podStartSLOduration=3.161376293 podStartE2EDuration="3.161376293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:16.300596949 +0000 UTC m=+777.373553933" watchObservedRunningTime="2026-10-29 21:59:18.161376293 +0000 UTC m=+779.234333256"
Oct 29 21:59:18.258773 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:18.258743    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 21:59:18.526308 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:18.522735    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 29 21:59:18.526308 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:18.522798    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh" podUID="f06b758b-d09f-4882-a198-d88e91c7f390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 29 21:59:19.520775 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:19.520742    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 29 21:59:19.520949 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:19.520797    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh" podUID="f06b758b-d09f-4882-a198-d88e91c7f390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 29 21:59:20.521351 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:20.521314    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 29 21:59:20.521745 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:20.521371    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh" podUID="f06b758b-d09f-4882-a198-d88e91c7f390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 29 21:59:21.520364 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:21.520322    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 29 21:59:21.520534 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:21.520392    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh" podUID="f06b758b-d09f-4882-a198-d88e91c7f390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 29 21:59:22.520598 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:22.520560    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 29 21:59:22.521020 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:22.520612    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh" podUID="f06b758b-d09f-4882-a198-d88e91c7f390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 29 21:59:22.826439 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:22.826346    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"]
Oct 29 21:59:22.830942 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:22.830923    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:22.861663 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:22.861636    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"]
Oct 29 21:59:22.938538 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:22.938506    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:22.938538 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:22.938540    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:22.938702 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:22.938564    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhwwd\" (UniqueName: \"kubernetes.io/projected/590ad9cd-3237-4783-8dea-82e67274b580-kube-api-access-mhwwd\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:22.938743 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:22.938727    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/590ad9cd-3237-4783-8dea-82e67274b580-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:22.938777 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:22.938756    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590ad9cd-3237-4783-8dea-82e67274b580-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:22.938843 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:22.938805    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:22.938888 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:22.938859    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:22.938925 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:22.938908    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590ad9cd-3237-4783-8dea-82e67274b580-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:22.938995 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:22.938983    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:23.040027 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.039990    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:23.040182 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.040163    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:23.040232 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.040195    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:23.040315 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.040237    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhwwd\" (UniqueName: \"kubernetes.io/projected/590ad9cd-3237-4783-8dea-82e67274b580-kube-api-access-mhwwd\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:23.040439 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.040382    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/590ad9cd-3237-4783-8dea-82e67274b580-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:23.040439 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.040414    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590ad9cd-3237-4783-8dea-82e67274b580-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:23.040564 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.040437    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:23.040564 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.040552    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:23.040663 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.040584    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:23.040663 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.040592    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:23.040663 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.040653    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590ad9cd-3237-4783-8dea-82e67274b580-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:23.040877 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.040855    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:23.040980 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.040963    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:23.041234 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.041206    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590ad9cd-3237-4783-8dea-82e67274b580-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:23.042635 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.042614    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:23.043100 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.043079    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590ad9cd-3237-4783-8dea-82e67274b580-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:23.063158 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.063131    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/590ad9cd-3237-4783-8dea-82e67274b580-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:23.063261 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.063192    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhwwd\" (UniqueName: \"kubernetes.io/projected/590ad9cd-3237-4783-8dea-82e67274b580-kube-api-access-mhwwd\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:23.147367 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.146546    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:23.582623 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.582592    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"]
Oct 29 21:59:23.583188 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:59:23.583159    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod590ad9cd_3237_4783_8dea_82e67274b580.slice/crio-91531365cac88323750addb04f5b011dc313d5b61d846b8da9af74f0f0543a26 WatchSource:0}: Error finding container 91531365cac88323750addb04f5b011dc313d5b61d846b8da9af74f0f0543a26: Status 404 returned error can't find the container with id 91531365cac88323750addb04f5b011dc313d5b61d846b8da9af74f0f0543a26
Oct 29 21:59:23.585546 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.585504    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:59:23.585661 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.585576    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:59:23.585661 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.585604    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:59:23.810486 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.810456    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:23.810931 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.810897    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:23.812189 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:23.812159    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 21:59:24.309970 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:24.309934    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j" event={"ID":"590ad9cd-3237-4783-8dea-82e67274b580","Type":"ContainerStarted","Data":"51b7485dab3d195f50fcae441fe893a75227dd05a3c12d77eb5b082648641ff5"}
Oct 29 21:59:24.310154 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:24.309992    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j" event={"ID":"590ad9cd-3237-4783-8dea-82e67274b580","Type":"ContainerStarted","Data":"91531365cac88323750addb04f5b011dc313d5b61d846b8da9af74f0f0543a26"}
Oct 29 21:59:24.364568 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:24.364523    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j" podStartSLOduration=2.364509618 podStartE2EDuration="2.364509618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:24.362447043 +0000 UTC m=+785.435404060" watchObservedRunningTime="2026-10-29 21:59:24.364509618 +0000 UTC m=+785.437466582"
Oct 29 21:59:25.147770 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:25.147734    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:25.149155 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:25.149131    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 29 21:59:25.149271 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:25.149179    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j" podUID="590ad9cd-3237-4783-8dea-82e67274b580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 29 21:59:26.147966 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:26.147933    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 29 21:59:26.148363 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:26.147986    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j" podUID="590ad9cd-3237-4783-8dea-82e67274b580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 29 21:59:27.148074 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:27.148041    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 29 21:59:27.148539 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:27.148098    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j" podUID="590ad9cd-3237-4783-8dea-82e67274b580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 29 21:59:28.147992 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:28.147960    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 29 21:59:28.148195 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:28.148037    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j" podUID="590ad9cd-3237-4783-8dea-82e67274b580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 29 21:59:29.148183 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:29.148148    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 29 21:59:29.148559 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:29.148202    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j" podUID="590ad9cd-3237-4783-8dea-82e67274b580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 29 21:59:30.148586 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:30.147318    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 29 21:59:30.148586 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:30.147386    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j" podUID="590ad9cd-3237-4783-8dea-82e67274b580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 29 21:59:31.151799 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:31.151769    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:31.152239 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:31.152163    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:31.152892 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:31.152876    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 21:59:36.885313 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:36.866377    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 21:59:36.914311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:36.902316    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q" podUID="58be6b18-3bda-47b5-8c32-42e0b2e87fc1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:36.984310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:36.983387    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:37.118481 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:37.118440    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 21:59:37.147221 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:37.146407    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:37.246533 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:37.246492    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:37.293940 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:37.293833    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 21:59:37.909907 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:37.909858    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q" podUID="58be6b18-3bda-47b5-8c32-42e0b2e87fc1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 29 21:59:37.911316 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:37.911271    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:59:38.041623 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.041584    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"]
Oct 29 21:59:38.059379 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.059350    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.153475 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.153431    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:38.221359 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.173633    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:59:38.313308 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.296362    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:38.313308 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.296957    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:59:38.313308 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.298087    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"]
Oct 29 21:59:38.516372 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.482500    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw4lq\" (UniqueName: \"kubernetes.io/projected/a437f4e2-dc10-41de-851a-ee488b08c6f4-kube-api-access-pw4lq\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.525573 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.525536    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.545307 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.537466    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.554458 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.554404    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.560743 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.560710    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a437f4e2-dc10-41de-851a-ee488b08c6f4-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.602469 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.602429    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.602845 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.602809    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.603258 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.603237    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.603475 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.603445    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.756339 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.755768    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:59:38.756339 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.755834    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:59:38.806359 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.802913    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.806359 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.802980    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.806359 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.803030    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw4lq\" (UniqueName: \"kubernetes.io/projected/a437f4e2-dc10-41de-851a-ee488b08c6f4-kube-api-access-pw4lq\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.806359 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.803107    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.806359 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.803194    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.806359 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.803222    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.806359 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.803304    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a437f4e2-dc10-41de-851a-ee488b08c6f4-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.806359 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.803379    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.806359 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.803435    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.806359 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.804817    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-credential-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.806359 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.805087    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-data\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.808031 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.808007    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-workload-socket\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.808568 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.808546    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-workload-certs\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.809675 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.809654    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a437f4e2-dc10-41de-851a-ee488b08c6f4-istiod-ca-cert\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.864114 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.864053    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:38.869533 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.866806    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:59:38.887966 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.886631    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw4lq\" (UniqueName: \"kubernetes.io/projected/a437f4e2-dc10-41de-851a-ee488b08c6f4-kube-api-access-pw4lq\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:38.942808 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.942242    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-token\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:39.003312 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:38.995804    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:39.209419 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:39.135227    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:39.209419 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:39.176063    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-podinfo\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:39.380634 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:39.380598    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-envoy\") pod \"e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") " pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:39.630369 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:39.627887    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:39.774144 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:39.774087    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:39.801345 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:39.800422    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q" podUID="58be6b18-3bda-47b5-8c32-42e0b2e87fc1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:39.897359 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:39.894908    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:40.001311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:40.000537    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:40.001311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:40.000977    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:59:40.021825 ip-10-0-129-146 kubenswrapper[4962]: E1029 21:59:40.020738    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 21:59:40.106312 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:40.100353    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q" podUID="58be6b18-3bda-47b5-8c32-42e0b2e87fc1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded"
Oct 29 21:59:40.150483 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:40.150379    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:40.214663 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:40.154225    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:59:40.779320 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:40.778862    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:40.825311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:40.804376    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q" podUID="58be6b18-3bda-47b5-8c32-42e0b2e87fc1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:40.825311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:40.805967    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:59:40.825311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:40.806033    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:59:40.853724 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:40.853353    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:40.980011 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:40.898068    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 21:59:40.980011 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:40.899505    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:59:41.003332 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:41.001075    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 21:59:41.161344 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:41.157995    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 21:59:41.320982 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:59:41.233852    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda437f4e2_dc10_41de_851a_ee488b08c6f4.slice/crio-34a9b53eba77b90358b87fe8e85198b3a1c4705ae6401fe42ce69e28a3eb000f WatchSource:0}: Error finding container 34a9b53eba77b90358b87fe8e85198b3a1c4705ae6401fe42ce69e28a3eb000f: Status 404 returned error can't find the container with id 34a9b53eba77b90358b87fe8e85198b3a1c4705ae6401fe42ce69e28a3eb000f
Oct 29 21:59:41.320982 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:41.258389    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:41.320982 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:41.310222    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:59:41.320982 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:41.310333    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:59:41.320982 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:41.310386    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:59:41.532319 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:41.517178    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 21:59:41.693778 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:41.693732    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"]
Oct 29 21:59:41.794477 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:41.794435    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 21:59:41.818384 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:41.814512    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:41.818384 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:41.814698    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q" podUID="58be6b18-3bda-47b5-8c32-42e0b2e87fc1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:41.866569 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:41.866528    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:41.870130 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:41.870106    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:59:41.899891 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:41.899539    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:41.945360 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:41.945331    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 21:59:41.962906 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:41.962881    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 21:59:42.020200 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:42.020158    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:42.119551 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:42.115403    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 21:59:42.194310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:42.142374    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 21:59:42.194310 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:42.142437    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg" podUID="16540bea-446b-46d9-9cde-4c9c34785ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 21:59:42.197240 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:42.196665    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 21:59:42.199481 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:42.198210    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 21:59:42.209755 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:42.209731    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 21:59:42.365046 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:42.365008    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5" event={"ID":"a437f4e2-dc10-41de-851a-ee488b08c6f4","Type":"ContainerStarted","Data":"edbdaac2be2e95304e218f52253c3354df3f1e301ecc6ae2fc886f2091923658"}
Oct 29 21:59:42.365896 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:42.365873    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5" event={"ID":"a437f4e2-dc10-41de-851a-ee488b08c6f4","Type":"ContainerStarted","Data":"34a9b53eba77b90358b87fe8e85198b3a1c4705ae6401fe42ce69e28a3eb000f"}
Oct 29 21:59:42.446028 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:42.445967    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5" podStartSLOduration=5.445948155 podStartE2EDuration="5.445948155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:42.445273322 +0000 UTC m=+803.518230312" watchObservedRunningTime="2026-10-29 21:59:42.445948155 +0000 UTC m=+803.518905119"
Oct 29 21:59:42.628889 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:42.628805    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:42.629976 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:42.629952    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 29 21:59:42.630089 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:42.630010    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5" podUID="a437f4e2-dc10-41de-851a-ee488b08c6f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 29 21:59:43.628932 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:43.628887    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 29 21:59:43.628932 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:43.628959    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5" podUID="a437f4e2-dc10-41de-851a-ee488b08c6f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 29 21:59:44.634311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:44.630474    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 29 21:59:44.634311 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:44.630541    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5" podUID="a437f4e2-dc10-41de-851a-ee488b08c6f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 29 21:59:45.629077 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:45.629039    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 29 21:59:45.629276 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:45.629105    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5" podUID="a437f4e2-dc10-41de-851a-ee488b08c6f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 29 21:59:46.629088 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:46.629053    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 29 21:59:46.629513 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:46.629107    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5" podUID="a437f4e2-dc10-41de-851a-ee488b08c6f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 29 21:59:47.628677 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:47.628645    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 29 21:59:47.628859 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:47.628695    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5" podUID="a437f4e2-dc10-41de-851a-ee488b08c6f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 29 21:59:48.629489 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:48.629448    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 29 21:59:48.629999 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:48.629499    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5" podUID="a437f4e2-dc10-41de-851a-ee488b08c6f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 29 21:59:49.687829 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:49.687788    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:49.688390 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:49.688334    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:49.690148 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:49.690122    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 21:59:54.229725 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.229690    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"]
Oct 29 21:59:54.235500 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.235482    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.266875 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.266847    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"]
Oct 29 21:59:54.271195 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.271172    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.278683 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.278156    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"]
Oct 29 21:59:54.291901 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.291876    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"]
Oct 29 21:59:54.399244 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.399211    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x7vt\" (UniqueName: \"kubernetes.io/projected/8054bc19-6db7-462f-8af0-335903ce62f3-kube-api-access-6x7vt\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.399445 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.399251    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.399445 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.399274    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.399445 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.399424    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/730edf47-8b07-477a-acfc-47faaaacfc84-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.399633 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.399463    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.399633 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.399535    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.399633 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.399570    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/730edf47-8b07-477a-acfc-47faaaacfc84-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.399774 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.399671    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sk4s\" (UniqueName: \"kubernetes.io/projected/730edf47-8b07-477a-acfc-47faaaacfc84-kube-api-access-4sk4s\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.399774 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.399706    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8054bc19-6db7-462f-8af0-335903ce62f3-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.399774 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.399730    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/730edf47-8b07-477a-acfc-47faaaacfc84-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.399881 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.399861    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.399913 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.399900    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8054bc19-6db7-462f-8af0-335903ce62f3-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.399959 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.399946    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.400001 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.399990    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.400068 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.400050    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.400174 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.400160    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.400224 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.400209    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.400318 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.400279    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8054bc19-6db7-462f-8af0-335903ce62f3-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.501501 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.501443    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.501617 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.501516    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.501617 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.501557    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.501617 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.501578    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8054bc19-6db7-462f-8af0-335903ce62f3-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.501617 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.501596    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x7vt\" (UniqueName: \"kubernetes.io/projected/8054bc19-6db7-462f-8af0-335903ce62f3-kube-api-access-6x7vt\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.501840 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.501617    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.501840 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.501644    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.501840 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.501678    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/730edf47-8b07-477a-acfc-47faaaacfc84-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.501840 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.501700    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.501840 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.501734    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.501840 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.501761    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/730edf47-8b07-477a-acfc-47faaaacfc84-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.501840 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.501824    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sk4s\" (UniqueName: \"kubernetes.io/projected/730edf47-8b07-477a-acfc-47faaaacfc84-kube-api-access-4sk4s\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.502206 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.501857    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8054bc19-6db7-462f-8af0-335903ce62f3-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.502206 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.501881    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/730edf47-8b07-477a-acfc-47faaaacfc84-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.502206 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.501903    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.502206 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.501912    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.502206 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.501953    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8054bc19-6db7-462f-8af0-335903ce62f3-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.502206 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.501998    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.502206 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.502016    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.502206 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.502064    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.502206 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.502080    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.502702 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.502255    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.502702 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.502369    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8054bc19-6db7-462f-8af0-335903ce62f3-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.502702 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.502447    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-credential-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.502702 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.502545    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-workload-socket\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.502921 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.502841    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-istio-data\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.502983 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.502943    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-workload-certs\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.503213 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.503188    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/730edf47-8b07-477a-acfc-47faaaacfc84-istiod-ca-cert\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.504127 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.504109    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.504353 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.504332    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-istio-envoy\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.504591 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.504576    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/730edf47-8b07-477a-acfc-47faaaacfc84-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.504940 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.504915    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8054bc19-6db7-462f-8af0-335903ce62f3-istio-podinfo\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.525322 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.525298    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8054bc19-6db7-462f-8af0-335903ce62f3-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.525571 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.525554    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x7vt\" (UniqueName: \"kubernetes.io/projected/8054bc19-6db7-462f-8af0-335903ce62f3-kube-api-access-6x7vt\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.525908 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.525887    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/730edf47-8b07-477a-acfc-47faaaacfc84-istio-token\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.525995 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.525967    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sk4s\" (UniqueName: \"kubernetes.io/projected/730edf47-8b07-477a-acfc-47faaaacfc84-kube-api-access-4sk4s\") pod \"e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") " pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.554209 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.554191    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:54.590563 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.590536    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:54.943510 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.943475    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"]
Oct 29 21:59:54.944025 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:59:54.943988    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730edf47_8b07_477a_acfc_47faaaacfc84.slice/crio-0d5703ca13a73adaef8685901e22556a82f0d7e2a46d561aa403c201a00b6c06 WatchSource:0}: Error finding container 0d5703ca13a73adaef8685901e22556a82f0d7e2a46d561aa403c201a00b6c06: Status 404 returned error can't find the container with id 0d5703ca13a73adaef8685901e22556a82f0d7e2a46d561aa403c201a00b6c06
Oct 29 21:59:54.946739 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.946708    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:59:54.946841 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.946771    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:59:54.946841 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.946797    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:59:54.947010 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.946986    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"]
Oct 29 21:59:54.959109 ip-10-0-129-146 kubenswrapper[4962]: W1029 21:59:54.959079    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8054bc19_6db7_462f_8af0_335903ce62f3.slice/crio-7b98fe2bd31c7f17e7b3af8c88838adf091090cda4bde2fe4c3403e7c2252f09 WatchSource:0}: Error finding container 7b98fe2bd31c7f17e7b3af8c88838adf091090cda4bde2fe4c3403e7c2252f09: Status 404 returned error can't find the container with id 7b98fe2bd31c7f17e7b3af8c88838adf091090cda4bde2fe4c3403e7c2252f09
Oct 29 21:59:54.961978 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.961943    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:59:54.962072 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.962032    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:59:54.962072 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:54.962066    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 21:59:55.475597 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:55.475556    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt" event={"ID":"730edf47-8b07-477a-acfc-47faaaacfc84","Type":"ContainerStarted","Data":"8f72340e66bc196b63adb77b7aeb06119281c2d3ecb65558ef9a6a315f89fbe4"}
Oct 29 21:59:55.476050 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:55.475622    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt" event={"ID":"730edf47-8b07-477a-acfc-47faaaacfc84","Type":"ContainerStarted","Data":"0d5703ca13a73adaef8685901e22556a82f0d7e2a46d561aa403c201a00b6c06"}
Oct 29 21:59:55.477353 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:55.477325    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz" event={"ID":"8054bc19-6db7-462f-8af0-335903ce62f3","Type":"ContainerStarted","Data":"c69b96ab47e84cbe96c8a6c22e84043a9517fba53d146ed6885fbeb0e930af70"}
Oct 29 21:59:55.477465 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:55.477373    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz" event={"ID":"8054bc19-6db7-462f-8af0-335903ce62f3","Type":"ContainerStarted","Data":"7b98fe2bd31c7f17e7b3af8c88838adf091090cda4bde2fe4c3403e7c2252f09"}
Oct 29 21:59:55.522696 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:55.522639    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt" podStartSLOduration=1.5226221130000002 podStartE2EDuration="1.522622113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:55.521592353 +0000 UTC m=+816.594549334" watchObservedRunningTime="2026-10-29 21:59:55.522622113 +0000 UTC m=+816.595579078"
Oct 29 21:59:55.554372 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:55.554344    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 21:59:55.568190 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:55.568148    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz" podStartSLOduration=1.568136729 podStartE2EDuration="1.568136729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 21:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 21:59:55.566752498 +0000 UTC m=+816.639709462" watchObservedRunningTime="2026-10-29 21:59:55.568136729 +0000 UTC m=+816.641093692"
Oct 29 21:59:55.590707 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:55.590683    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 21:59:56.556074 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:56.556028    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 29 21:59:56.556461 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:56.556091    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt" podUID="730edf47-8b07-477a-acfc-47faaaacfc84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 29 21:59:56.608805 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:56.608757    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 29 21:59:56.609094 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:56.609049    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz" podUID="8054bc19-6db7-462f-8af0-335903ce62f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 29 21:59:57.554654 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:57.554625    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 29 21:59:57.554823 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:57.554676    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt" podUID="730edf47-8b07-477a-acfc-47faaaacfc84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 29 21:59:57.599317 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:57.593622    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 29 21:59:57.599317 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:57.593680    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz" podUID="8054bc19-6db7-462f-8af0-335903ce62f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 29 21:59:57.663250 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:57.663220    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"]
Oct 29 21:59:58.555123 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:58.555089    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 29 21:59:58.555314 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:58.555140    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt" podUID="730edf47-8b07-477a-acfc-47faaaacfc84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 29 21:59:58.597654 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:58.597619    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 29 21:59:58.597897 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:58.597871    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz" podUID="8054bc19-6db7-462f-8af0-335903ce62f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 29 21:59:59.555177 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:59.555145    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 29 21:59:59.555665 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:59.555407    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt" podUID="730edf47-8b07-477a-acfc-47faaaacfc84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 29 21:59:59.600422 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:59.600386    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 29 21:59:59.600718 ip-10-0-129-146 kubenswrapper[4962]: I1029 21:59:59.600692    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz" podUID="8054bc19-6db7-462f-8af0-335903ce62f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 29 22:00:00.166027 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.165995    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8"]
Oct 29 22:00:00.170634 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.170614    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8"
Oct 29 22:00:00.239045 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.214470    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:00:00.239045 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.237502    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d"]
Oct 29 22:00:00.243061 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.243037    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d"
Oct 29 22:00:00.256432 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.255984    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8"]
Oct 29 22:00:00.257580 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.257558    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d"]
Oct 29 22:00:00.339236 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.339193    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8765f54-99e8-4498-9591-2a7a1ca0059e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d\" (UID: \"c8765f54-99e8-4498-9591-2a7a1ca0059e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d"
Oct 29 22:00:00.339499 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.339478    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6fcbf03-5017-48ca-a294-7b837a9d8877-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8\" (UID: \"e6fcbf03-5017-48ca-a294-7b837a9d8877\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8"
Oct 29 22:00:00.339822 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.339805    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5kxm\" (UniqueName: \"kubernetes.io/projected/c8765f54-99e8-4498-9591-2a7a1ca0059e-kube-api-access-f5kxm\") pod \"maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d\" (UID: \"c8765f54-99e8-4498-9591-2a7a1ca0059e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d"
Oct 29 22:00:00.339961 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.339946    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ttxf\" (UniqueName: \"kubernetes.io/projected/e6fcbf03-5017-48ca-a294-7b837a9d8877-kube-api-access-6ttxf\") pod \"maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8\" (UID: \"e6fcbf03-5017-48ca-a294-7b837a9d8877\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8"
Oct 29 22:00:00.352637 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.352610    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"]
Oct 29 22:00:00.357324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.357307    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.436380 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.433813    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"]
Oct 29 22:00:00.444037 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.444003    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5kxm\" (UniqueName: \"kubernetes.io/projected/c8765f54-99e8-4498-9591-2a7a1ca0059e-kube-api-access-f5kxm\") pod \"maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d\" (UID: \"c8765f54-99e8-4498-9591-2a7a1ca0059e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d"
Oct 29 22:00:00.444309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.444270    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ttxf\" (UniqueName: \"kubernetes.io/projected/e6fcbf03-5017-48ca-a294-7b837a9d8877-kube-api-access-6ttxf\") pod \"maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8\" (UID: \"e6fcbf03-5017-48ca-a294-7b837a9d8877\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8"
Oct 29 22:00:00.444456 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.444436    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8765f54-99e8-4498-9591-2a7a1ca0059e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d\" (UID: \"c8765f54-99e8-4498-9591-2a7a1ca0059e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d"
Oct 29 22:00:00.444604 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.444587    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6fcbf03-5017-48ca-a294-7b837a9d8877-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8\" (UID: \"e6fcbf03-5017-48ca-a294-7b837a9d8877\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8"
Oct 29 22:00:00.445431 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.445406    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6fcbf03-5017-48ca-a294-7b837a9d8877-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8\" (UID: \"e6fcbf03-5017-48ca-a294-7b837a9d8877\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8"
Oct 29 22:00:00.445529 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.445503    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8765f54-99e8-4498-9591-2a7a1ca0059e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d\" (UID: \"c8765f54-99e8-4498-9591-2a7a1ca0059e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d"
Oct 29 22:00:00.469792 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.469747    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5kxm\" (UniqueName: \"kubernetes.io/projected/c8765f54-99e8-4498-9591-2a7a1ca0059e-kube-api-access-f5kxm\") pod \"maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d\" (UID: \"c8765f54-99e8-4498-9591-2a7a1ca0059e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d"
Oct 29 22:00:00.470445 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.470425    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ttxf\" (UniqueName: \"kubernetes.io/projected/e6fcbf03-5017-48ca-a294-7b837a9d8877-kube-api-access-6ttxf\") pod \"maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8\" (UID: \"e6fcbf03-5017-48ca-a294-7b837a9d8877\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8"
Oct 29 22:00:00.546203 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.546169    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a71a069-94f9-4736-8b9f-d43292554254-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.546488 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.546469    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a71a069-94f9-4736-8b9f-d43292554254-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.546612 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.546599    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a71a069-94f9-4736-8b9f-d43292554254-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.546718 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.546703    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.546901 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.546883    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8"
Oct 29 22:00:00.547005 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.546989    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.547168 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.547153    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.547259 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.547249    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.547364 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.547353    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9mtz\" (UniqueName: \"kubernetes.io/projected/2a71a069-94f9-4736-8b9f-d43292554254-kube-api-access-k9mtz\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.547444 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.547435    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.554798 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.554774    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:00:00.554889 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.554819    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt" podUID="730edf47-8b07-477a-acfc-47faaaacfc84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 29 22:00:00.571011 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.570987    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d"
Oct 29 22:00:00.626458 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.618467    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:00:00.626458 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.618521    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz" podUID="8054bc19-6db7-462f-8af0-335903ce62f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 29 22:00:00.650126 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.649160    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.650126 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.649276    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.650126 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.649348    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.650126 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.649384    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9mtz\" (UniqueName: \"kubernetes.io/projected/2a71a069-94f9-4736-8b9f-d43292554254-kube-api-access-k9mtz\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.650126 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.649428    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.650126 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.649511    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a71a069-94f9-4736-8b9f-d43292554254-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.650126 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.649584    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a71a069-94f9-4736-8b9f-d43292554254-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.650126 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.649628    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a71a069-94f9-4736-8b9f-d43292554254-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.650126 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.649653    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.651448 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.650893    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-credential-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.651448 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.651088    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-workload-socket\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.651448 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.651136    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-workload-certs\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.651448 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.651406    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-istio-data\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.651788 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.651768    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a71a069-94f9-4736-8b9f-d43292554254-istiod-ca-cert\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.652769 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.652749    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-istio-envoy\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.653732 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.653709    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a71a069-94f9-4736-8b9f-d43292554254-istio-podinfo\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.736535 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.736456    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a71a069-94f9-4736-8b9f-d43292554254-istio-token\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.745065 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.745016    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9mtz\" (UniqueName: \"kubernetes.io/projected/2a71a069-94f9-4736-8b9f-d43292554254-kube-api-access-k9mtz\") pod \"e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") " pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:00.923487 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.923445    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8"]
Oct 29 22:00:00.930706 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:00:00.930672    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6fcbf03_5017_48ca_a294_7b837a9d8877.slice/crio-7727b625b111311a7de8c1d102a4dce0655626fe9bf9238a7a73d563438551ed WatchSource:0}: Error finding container 7727b625b111311a7de8c1d102a4dce0655626fe9bf9238a7a73d563438551ed: Status 404 returned error can't find the container with id 7727b625b111311a7de8c1d102a4dce0655626fe9bf9238a7a73d563438551ed
Oct 29 22:00:00.934245 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:00.934230    4962 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:00:01.025325 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:01.025240    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:01.025692 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:01.025666    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d"]
Oct 29 22:00:01.027986 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:00:01.027962    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8765f54_99e8_4498_9591_2a7a1ca0059e.slice/crio-93f3fbcfd4af33fae0186f7eb4aa7f335cb95b57000287bd9fb0e1e37e17ba24 WatchSource:0}: Error finding container 93f3fbcfd4af33fae0186f7eb4aa7f335cb95b57000287bd9fb0e1e37e17ba24: Status 404 returned error can't find the container with id 93f3fbcfd4af33fae0186f7eb4aa7f335cb95b57000287bd9fb0e1e37e17ba24
Oct 29 22:00:01.438307 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:01.434037    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"]
Oct 29 22:00:01.478394 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:00:01.468371    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a71a069_94f9_4736_8b9f_d43292554254.slice/crio-9173e0d0267144bf05845444c74b2cd68fa38d6395c6d2c9f94fec8fb8502442 WatchSource:0}: Error finding container 9173e0d0267144bf05845444c74b2cd68fa38d6395c6d2c9f94fec8fb8502442: Status 404 returned error can't find the container with id 9173e0d0267144bf05845444c74b2cd68fa38d6395c6d2c9f94fec8fb8502442
Oct 29 22:00:01.562805 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:01.532324    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:00:01.562805 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:01.532434    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:00:01.562805 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:01.532478    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:00:01.573149 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:01.569751    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 29 22:00:01.573149 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:01.569810    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt" podUID="730edf47-8b07-477a-acfc-47faaaacfc84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 29 22:00:01.593779 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:01.593749    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 29 22:00:01.593977 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:01.593954    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz" podUID="8054bc19-6db7-462f-8af0-335903ce62f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 29 22:00:01.599539 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:01.598170    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d" event={"ID":"c8765f54-99e8-4498-9591-2a7a1ca0059e","Type":"ContainerStarted","Data":"93f3fbcfd4af33fae0186f7eb4aa7f335cb95b57000287bd9fb0e1e37e17ba24"}
Oct 29 22:00:01.600185 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:01.600104    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds" event={"ID":"2a71a069-94f9-4736-8b9f-d43292554254","Type":"ContainerStarted","Data":"9173e0d0267144bf05845444c74b2cd68fa38d6395c6d2c9f94fec8fb8502442"}
Oct 29 22:00:01.601925 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:01.601770    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8" event={"ID":"e6fcbf03-5017-48ca-a294-7b837a9d8877","Type":"ContainerStarted","Data":"7727b625b111311a7de8c1d102a4dce0655626fe9bf9238a7a73d563438551ed"}
Oct 29 22:00:02.652190 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:02.652144    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds" event={"ID":"2a71a069-94f9-4736-8b9f-d43292554254","Type":"ContainerStarted","Data":"dcf87bbef5ce6b2f7eaaaa0ab0fdfa34fe4bf44ac593049d59c0c42307689d6d"}
Oct 29 22:00:02.794309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:02.768997    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds" podStartSLOduration=2.768978153 podStartE2EDuration="2.768978153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:02.736742597 +0000 UTC m=+823.809699583" watchObservedRunningTime="2026-10-29 22:00:02.768978153 +0000 UTC m=+823.841935117"
Oct 29 22:00:03.025742 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:03.025660    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:03.035993 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:03.035960    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:00:03.036224 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:03.036194    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds" podUID="2a71a069-94f9-4736-8b9f-d43292554254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 29 22:00:03.158883 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:03.158841    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 22:00:03.159622 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:03.159597    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 22:00:03.164766 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:03.164731    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 22:00:03.166209 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:03.166186    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 22:00:03.170704 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:03.167387    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 22:00:03.170704 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:03.169852    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 22:00:04.026019 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.025666    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:00:04.026019 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.025719    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds" podUID="2a71a069-94f9-4736-8b9f-d43292554254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 29 22:00:04.162654 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.162612    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"]
Oct 29 22:00:04.169204 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.169175    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.232989 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.231416    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"]
Oct 29 22:00:04.233763 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.233739    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.234084 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.234064    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.234376 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.234356    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.234583 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.234565    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.234722 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.234707    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9xj8\" (UniqueName: \"kubernetes.io/projected/67b4a685-bfa5-4a36-adf7-68c7591137c3-kube-api-access-j9xj8\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.234853 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.234839    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.235401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.235385    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b4a685-bfa5-4a36-adf7-68c7591137c3-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.235564 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.235550    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.235662 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.235652    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.240096 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.239631    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"]
Oct 29 22:00:04.246574 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.246558    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.258877 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.257818    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"]
Oct 29 22:00:04.337393 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.337282    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5e71cf-5078-4917-94b8-2b531b143393-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.337393 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.337377    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.337620 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.337456    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.337893 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.337761    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9xj8\" (UniqueName: \"kubernetes.io/projected/67b4a685-bfa5-4a36-adf7-68c7591137c3-kube-api-access-j9xj8\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.337893 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.337819    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.337893 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.337835    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.337893 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.337848    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5e71cf-5078-4917-94b8-2b531b143393-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.339398 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.337938    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.339398 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.338001    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.339398 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.338031    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.339398 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.338063    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b4a685-bfa5-4a36-adf7-68c7591137c3-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.339398 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.338369    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5e71cf-5078-4917-94b8-2b531b143393-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.339398 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.338399    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.339398 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.338434    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbzxf\" (UniqueName: \"kubernetes.io/projected/5d5e71cf-5078-4917-94b8-2b531b143393-kube-api-access-zbzxf\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.339398 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.338477    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.339398 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.338523    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.339398 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.338577    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.339398 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.338603    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.339398 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.338632    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.339398 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.338794    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b4a685-bfa5-4a36-adf7-68c7591137c3-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.339398 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.338958    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.339398 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.339049    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.339398 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.339327    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.341582 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.341554    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.342422 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.342404    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.361039 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.361016    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9xj8\" (UniqueName: \"kubernetes.io/projected/67b4a685-bfa5-4a36-adf7-68c7591137c3-kube-api-access-j9xj8\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.361477 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.361455    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.439498 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.439468    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.439637 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.439508    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5e71cf-5078-4917-94b8-2b531b143393-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.439637 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.439576    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.439637 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.439606    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.439743 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.439703    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5e71cf-5078-4917-94b8-2b531b143393-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.439777 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.439740    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbzxf\" (UniqueName: \"kubernetes.io/projected/5d5e71cf-5078-4917-94b8-2b531b143393-kube-api-access-zbzxf\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.439821 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.439804    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.439883 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.439871    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.439924 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.439891    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.439957 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.439920    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5e71cf-5078-4917-94b8-2b531b143393-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.440856 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.440830    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5e71cf-5078-4917-94b8-2b531b143393-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.441074 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.441055    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.441270 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.441253    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.442031 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.442010    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.444279 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.444235    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5e71cf-5078-4917-94b8-2b531b143393-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.445867 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.445812    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.464541 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.464517    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5e71cf-5078-4917-94b8-2b531b143393-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.465511 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.465485    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbzxf\" (UniqueName: \"kubernetes.io/projected/5d5e71cf-5078-4917-94b8-2b531b143393-kube-api-access-zbzxf\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.535671 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.535640    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:04.566008 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.565834    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:04.910519 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.910483    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"]
Oct 29 22:00:04.912134 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:00:04.912107    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b4a685_bfa5_4a36_adf7_68c7591137c3.slice/crio-266454f7998a7b81c9eb10f1a66075ae36f3840edda628f4cc8146cfa6d59be3 WatchSource:0}: Error finding container 266454f7998a7b81c9eb10f1a66075ae36f3840edda628f4cc8146cfa6d59be3: Status 404 returned error can't find the container with id 266454f7998a7b81c9eb10f1a66075ae36f3840edda628f4cc8146cfa6d59be3
Oct 29 22:00:04.919579 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.919542    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:00:04.919763 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.919741    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:00:04.919964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.919943    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:00:04.960948 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.960920    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"]
Oct 29 22:00:04.963101 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:00:04.963071    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5e71cf_5078_4917_94b8_2b531b143393.slice/crio-e34b36b425dc454b9b545a2b456ab70f74cf06c4d4403e877db11d5d7f56be6a WatchSource:0}: Error finding container e34b36b425dc454b9b545a2b456ab70f74cf06c4d4403e877db11d5d7f56be6a: Status 404 returned error can't find the container with id e34b36b425dc454b9b545a2b456ab70f74cf06c4d4403e877db11d5d7f56be6a
Oct 29 22:00:04.967257 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.966786    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:00:04.967257 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.966864    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:00:04.967257 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:04.966903    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:00:05.027351 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.025736    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:00:05.027351 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.025782    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds" podUID="2a71a069-94f9-4736-8b9f-d43292554254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 29 22:00:05.451559 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.451525    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"]
Oct 29 22:00:05.467648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.467618    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.519569 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.501759    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"]
Oct 29 22:00:05.648010 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.609755    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.648010 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.609824    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf79q\" (UniqueName: \"kubernetes.io/projected/295e3d96-ff42-4be3-b4de-cd85ca4d8578-kube-api-access-wf79q\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.648010 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.609904    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.648010 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.610064    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.648010 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.610092    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.648010 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.610132    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.648010 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.610167    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.648010 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.610300    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.648010 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.610365    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.711782 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.711694    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.711955 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.711828    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.711955 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.711857    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.711955 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.711880    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.711955 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.711911    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.712185 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.712001    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.712185 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.712045    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.712309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.712226    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.712309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.712262    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf79q\" (UniqueName: \"kubernetes.io/projected/295e3d96-ff42-4be3-b4de-cd85ca4d8578-kube-api-access-wf79q\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.713257 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.713225    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.713872 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.713843    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.714096 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.714069    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.714096 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.714083    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.715227 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.714319    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.719660 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.717152    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.720099 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.720039    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.724852 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.723743    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp" event={"ID":"5d5e71cf-5078-4917-94b8-2b531b143393","Type":"ContainerStarted","Data":"3c7774bee68cad1eae65e008896454a04da85c5ea27d3083c5f4af47e55854ad"}
Oct 29 22:00:05.724852 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.723803    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp" event={"ID":"5d5e71cf-5078-4917-94b8-2b531b143393","Type":"ContainerStarted","Data":"e34b36b425dc454b9b545a2b456ab70f74cf06c4d4403e877db11d5d7f56be6a"}
Oct 29 22:00:05.729645 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.728815    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57" event={"ID":"67b4a685-bfa5-4a36-adf7-68c7591137c3","Type":"ContainerStarted","Data":"733129bde7f2b2532aaf9c3791293ff0e39979c9dafb1975650ec8f7f4bdd8fd"}
Oct 29 22:00:05.729645 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.728871    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57" event={"ID":"67b4a685-bfa5-4a36-adf7-68c7591137c3","Type":"ContainerStarted","Data":"266454f7998a7b81c9eb10f1a66075ae36f3840edda628f4cc8146cfa6d59be3"}
Oct 29 22:00:05.738758 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.738671    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.748576 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.748537    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf79q\" (UniqueName: \"kubernetes.io/projected/295e3d96-ff42-4be3-b4de-cd85ca4d8578-kube-api-access-wf79q\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:05.800349 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.787851    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp" podStartSLOduration=1.787819228 podStartE2EDuration="1.787819228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:05.782952472 +0000 UTC m=+826.855909433" watchObservedRunningTime="2026-10-29 22:00:05.787819228 +0000 UTC m=+826.860776189"
Oct 29 22:00:05.848309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:05.846737    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57" podStartSLOduration=1.846717183 podStartE2EDuration="1.846717183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:05.84471046 +0000 UTC m=+826.917667425" watchObservedRunningTime="2026-10-29 22:00:05.846717183 +0000 UTC m=+826.919674148"
Oct 29 22:00:06.030247 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:06.029698    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:00:06.030247 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:06.029789    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds" podUID="2a71a069-94f9-4736-8b9f-d43292554254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 29 22:00:06.032364 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:06.032335    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:06.548367 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:06.544637    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:06.548367 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:06.544769    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:00:06.548367 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:06.544813    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57" podUID="67b4a685-bfa5-4a36-adf7-68c7591137c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 29 22:00:06.568051 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:06.567009    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:06.568184 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:06.568090    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:00:06.568184 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:06.568129    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp" podUID="5d5e71cf-5078-4917-94b8-2b531b143393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 29 22:00:06.864889 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:06.864847    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:00:06.865131 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:06.865101    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:00:06.865268 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:06.865250    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:00:06.865914 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:06.865890    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"]
Oct 29 22:00:07.035900 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:07.035656    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:00:07.035900 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:07.035712    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds" podUID="2a71a069-94f9-4736-8b9f-d43292554254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 29 22:00:07.790314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:07.787811    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:00:07.790314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:07.787881    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57" podUID="67b4a685-bfa5-4a36-adf7-68c7591137c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 29 22:00:07.790314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:07.788706    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:00:07.790314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:07.788749    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp" podUID="5d5e71cf-5078-4917-94b8-2b531b143393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 29 22:00:08.011577 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:08.011539    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6" event={"ID":"295e3d96-ff42-4be3-b4de-cd85ca4d8578","Type":"ContainerStarted","Data":"755a669af2f3c96b2ec3eec0dc1d9c2726cc8f9205c8cc4f080534a9a6684958"}
Oct 29 22:00:08.011877 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:08.011851    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6" event={"ID":"295e3d96-ff42-4be3-b4de-cd85ca4d8578","Type":"ContainerStarted","Data":"a8f80e2cfc0567a627f4e74c7f93478e7b2d094e7ca7408ca60923b63fcd8a9d"}
Oct 29 22:00:08.032609 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:08.032563    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:00:08.032917 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:08.032865    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds" podUID="2a71a069-94f9-4736-8b9f-d43292554254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 29 22:00:08.033783 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:08.033765    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:08.039462 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:08.039438    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:00:08.039956 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:08.039928    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6" podUID="295e3d96-ff42-4be3-b4de-cd85ca4d8578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 29 22:00:08.107363 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:08.106489    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6" podStartSLOduration=3.106470727 podStartE2EDuration="3.106470727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:08.081948833 +0000 UTC m=+829.154905834" watchObservedRunningTime="2026-10-29 22:00:08.106470727 +0000 UTC m=+829.179427693"
Oct 29 22:00:08.537946 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:08.537911    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:00:08.538253 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:08.538208    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57" podUID="67b4a685-bfa5-4a36-adf7-68c7591137c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 29 22:00:08.567313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:08.566771    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:00:08.567313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:08.566846    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp" podUID="5d5e71cf-5078-4917-94b8-2b531b143393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 29 22:00:09.043412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:09.034637    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 29 22:00:09.043412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:09.034701    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds" podUID="2a71a069-94f9-4736-8b9f-d43292554254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 29 22:00:09.051321 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:09.050795    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:00:09.051321 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:09.050864    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6" podUID="295e3d96-ff42-4be3-b4de-cd85ca4d8578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 29 22:00:09.539423 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:09.539386    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:00:09.544794 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:09.544748    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57" podUID="67b4a685-bfa5-4a36-adf7-68c7591137c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 29 22:00:09.574381 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:09.567029    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:00:09.574381 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:09.567098    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp" podUID="5d5e71cf-5078-4917-94b8-2b531b143393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 29 22:00:10.012330 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:00:10.012253    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:00:10.035028 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:10.034995    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:00:10.035301 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:10.035259    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6" podUID="295e3d96-ff42-4be3-b4de-cd85ca4d8578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 29 22:00:10.417480 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:10.417421    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:10.523829 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:10.523770    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:10.545032 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:10.545001    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:00:10.545309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:10.545256    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57" podUID="67b4a685-bfa5-4a36-adf7-68c7591137c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 29 22:00:10.574360 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:10.574329    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:00:10.574588 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:10.574561    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp" podUID="5d5e71cf-5078-4917-94b8-2b531b143393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 29 22:00:10.658957 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:10.658923    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:10.660871 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:10.660848    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:10.666944 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:10.666918    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:00:11.262960 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:11.262897    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:00:11.263181 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:11.262964    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:00:11.264161 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:11.263244    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:00:11.264161 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:11.263279    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6" podUID="295e3d96-ff42-4be3-b4de-cd85ca4d8578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 29 22:00:11.341563 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:11.303855    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:11.350393 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:11.342928    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:11.350393 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:11.347928    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:00:11.876362 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:11.852406    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:00:11.876362 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:11.852483    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57" podUID="67b4a685-bfa5-4a36-adf7-68c7591137c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 29 22:00:11.876362 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:11.853835    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:00:11.876362 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:11.854040    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp" podUID="5d5e71cf-5078-4917-94b8-2b531b143393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 29 22:00:11.946448 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:11.946405    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:00:12.036500 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:12.036444    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:00:12.036686 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:12.036522    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6" podUID="295e3d96-ff42-4be3-b4de-cd85ca4d8578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 29 22:00:12.536583 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:12.536542    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 29 22:00:12.536774 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:12.536611    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57" podUID="67b4a685-bfa5-4a36-adf7-68c7591137c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 29 22:00:12.567164 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:12.567132    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 29 22:00:12.567314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:12.567189    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp" podUID="5d5e71cf-5078-4917-94b8-2b531b143393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 29 22:00:12.731810 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:12.731768    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q" podUID="58be6b18-3bda-47b5-8c32-42e0b2e87fc1" containerName="main" containerID="cri-o://d21e243c987fcc0d35e494bff9bc3d6e0c7032e69def21552db363500c0244aa" gracePeriod=45
Oct 29 22:00:13.068804 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:13.066209    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:00:13.068804 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:13.066269    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6" podUID="295e3d96-ff42-4be3-b4de-cd85ca4d8578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 29 22:00:13.344586 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:13.344549    4962 generic.go:356] "Generic (PLEG): container finished" podID="58be6b18-3bda-47b5-8c32-42e0b2e87fc1" containerID="d21e243c987fcc0d35e494bff9bc3d6e0c7032e69def21552db363500c0244aa" exitCode=0
Oct 29 22:00:13.362361 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:13.362326    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q" event={"ID":"58be6b18-3bda-47b5-8c32-42e0b2e87fc1","Type":"ContainerDied","Data":"d21e243c987fcc0d35e494bff9bc3d6e0c7032e69def21552db363500c0244aa"}
Oct 29 22:00:13.378472 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:13.378443    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 22:00:13.508809 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:13.506204    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-dshm\") pod \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\" (UID: \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\") "
Oct 29 22:00:13.508809 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:13.506333    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-tls-certs\" (UniqueName: \"kubernetes.io/secret/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-tls-certs\") pod \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\" (UID: \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\") "
Oct 29 22:00:13.508809 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:13.506364    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-home\" (UniqueName: \"kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-home\") pod \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\" (UID: \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\") "
Oct 29 22:00:13.508809 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:13.506432    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-kube-api-access-82h5t\" (UniqueName: \"kubernetes.io/projected/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-kube-api-access-82h5t\") pod \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\" (UID: \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\") "
Oct 29 22:00:13.508809 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:13.506608    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-tmp-dir\") pod \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\" (UID: \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\") "
Oct 29 22:00:13.508809 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:13.506712    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-model-cache\") pod \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\" (UID: \"58be6b18-3bda-47b5-8c32-42e0b2e87fc1\") "
Oct 29 22:00:13.520902 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:13.520852    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-tmp-dir" pod "58be6b18-3bda-47b5-8c32-42e0b2e87fc1" (UID: "58be6b18-3bda-47b5-8c32-42e0b2e87fc1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:00:13.524307 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:13.522530    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-model-cache" pod "58be6b18-3bda-47b5-8c32-42e0b2e87fc1" (UID: "58be6b18-3bda-47b5-8c32-42e0b2e87fc1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:00:13.524307 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:13.523410    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-home" pod "58be6b18-3bda-47b5-8c32-42e0b2e87fc1" (UID: "58be6b18-3bda-47b5-8c32-42e0b2e87fc1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:00:13.617266 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:13.615404    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-dshm" pod "58be6b18-3bda-47b5-8c32-42e0b2e87fc1" (UID: "58be6b18-3bda-47b5-8c32-42e0b2e87fc1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:00:13.617266 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:13.617187    4962 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-tmp-dir\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:00:13.617266 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:13.617214    4962 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-model-cache\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:00:13.617266 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:13.617228    4962 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-dshm\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:00:13.617266 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:13.617240    4962 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-home\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:00:13.628612 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:13.627414    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-tls-certs" pod "58be6b18-3bda-47b5-8c32-42e0b2e87fc1" (UID: "58be6b18-3bda-47b5-8c32-42e0b2e87fc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:00:13.628612 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:13.627526    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-kube-api-access-82h5t" pod "58be6b18-3bda-47b5-8c32-42e0b2e87fc1" (UID: "58be6b18-3bda-47b5-8c32-42e0b2e87fc1"). InnerVolumeSpecName "kube-api-access-82h5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:00:13.733538 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:13.726770    4962 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-tls-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:00:13.733538 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:13.726800    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82h5t\" (UniqueName: \"kubernetes.io/projected/58be6b18-3bda-47b5-8c32-42e0b2e87fc1-kube-api-access-82h5t\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:00:14.081377 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:14.039586    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:00:14.081377 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:14.039653    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6" podUID="295e3d96-ff42-4be3-b4de-cd85ca4d8578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 29 22:00:14.081377 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:14.070121    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:14.207391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:14.203532    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:14.217307 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:14.210692    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:00:14.222781 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:14.221103    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q" podUID="58be6b18-3bda-47b5-8c32-42e0b2e87fc1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:14.333502 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:14.316658    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:14.333502 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:14.327820    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:14.333502 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:14.329263    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:00:14.368342 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:14.368312    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8" event={"ID":"e6fcbf03-5017-48ca-a294-7b837a9d8877","Type":"ContainerStarted","Data":"a33c9a81c6a8593d05d61d337edfae4d45488f272af301daaf7390359fdb2f64"}
Oct 29 22:00:14.370770 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:14.370746    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"
Oct 29 22:00:14.371023 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:14.370845    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q" event={"ID":"58be6b18-3bda-47b5-8c32-42e0b2e87fc1","Type":"ContainerDied","Data":"ba595625a6884578e5f3e2c0d0bec6f681666577b905f122b8c13f130bf68ce3"}
Oct 29 22:00:14.371144 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:14.371132    4962 scope.go:122] "RemoveContainer" containerID="d21e243c987fcc0d35e494bff9bc3d6e0c7032e69def21552db363500c0244aa"
Oct 29 22:00:14.402912 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:14.402880    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d" event={"ID":"c8765f54-99e8-4498-9591-2a7a1ca0059e","Type":"ContainerStarted","Data":"398d593824e39e5c2433b9b314be3726331d0f527cfec92d2726ee9f49e19c36"}
Oct 29 22:00:14.509993 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:14.509883    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8" podStartSLOduration=2.1922808209999998 podStartE2EDuration="14.509863763s" totalImagesPullingTime="12.317582942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:14.507185929 +0000 UTC m=+835.580142895" watchObservedRunningTime="2026-10-29 22:00:14.509863763 +0000 UTC m=+835.582820729"
Oct 29 22:00:14.611720 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:14.611574    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d" podStartSLOduration=2.212617147 podStartE2EDuration="14.61155562s" totalImagesPullingTime="12.398938473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:14.553014985 +0000 UTC m=+835.625971949" watchObservedRunningTime="2026-10-29 22:00:14.61155562 +0000 UTC m=+835.684512585"
Oct 29 22:00:14.615005 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:14.614980    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"]
Oct 29 22:00:14.622137 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:14.622113    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/e2e-conflict-a-eab27cf7-kserve-65f9959f4b-w7n5q"]
Oct 29 22:00:15.033361 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:15.033318    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:00:15.033539 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:15.033390    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6" podUID="295e3d96-ff42-4be3-b4de-cd85ca4d8578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 29 22:00:15.063890 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:15.063859    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58be6b18-3bda-47b5-8c32-42e0b2e87fc1" path="/var/lib/kubelet/pods/58be6b18-3bda-47b5-8c32-42e0b2e87fc1/volumes"
Oct 29 22:00:15.412632 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:15.412597    4962 generic.go:356] "Generic (PLEG): container finished" podID="e6fcbf03-5017-48ca-a294-7b837a9d8877" containerID="a33c9a81c6a8593d05d61d337edfae4d45488f272af301daaf7390359fdb2f64" exitCode=60
Oct 29 22:00:15.413086 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:15.412726    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8" event={"ID":"e6fcbf03-5017-48ca-a294-7b837a9d8877","Type":"ContainerDied","Data":"a33c9a81c6a8593d05d61d337edfae4d45488f272af301daaf7390359fdb2f64"}
Oct 29 22:00:15.413086 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:15.413074    4962 scope.go:122] "RemoveContainer" containerID="a33c9a81c6a8593d05d61d337edfae4d45488f272af301daaf7390359fdb2f64"
Oct 29 22:00:15.418181 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:15.418157    4962 generic.go:356] "Generic (PLEG): container finished" podID="c8765f54-99e8-4498-9591-2a7a1ca0059e" containerID="398d593824e39e5c2433b9b314be3726331d0f527cfec92d2726ee9f49e19c36" exitCode=60
Oct 29 22:00:15.418319 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:15.418238    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d" event={"ID":"c8765f54-99e8-4498-9591-2a7a1ca0059e","Type":"ContainerDied","Data":"398d593824e39e5c2433b9b314be3726331d0f527cfec92d2726ee9f49e19c36"}
Oct 29 22:00:15.418588 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:15.418538    4962 scope.go:122] "RemoveContainer" containerID="398d593824e39e5c2433b9b314be3726331d0f527cfec92d2726ee9f49e19c36"
Oct 29 22:00:16.036956 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:16.036887    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:16.037274 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:16.037258    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:16.037941 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:16.037926    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:00:16.427791 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:16.427763    4962 generic.go:356] "Generic (PLEG): container finished" podID="e6fcbf03-5017-48ca-a294-7b837a9d8877" containerID="ca202cfcda423dd3e6c21f34129a1ba08b8259feef77f870b6770e93107b154c" exitCode=60
Oct 29 22:00:16.428173 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:16.427846    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8" event={"ID":"e6fcbf03-5017-48ca-a294-7b837a9d8877","Type":"ContainerDied","Data":"ca202cfcda423dd3e6c21f34129a1ba08b8259feef77f870b6770e93107b154c"}
Oct 29 22:00:16.428173 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:16.427921    4962 scope.go:122] "RemoveContainer" containerID="a33c9a81c6a8593d05d61d337edfae4d45488f272af301daaf7390359fdb2f64"
Oct 29 22:00:16.428279 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:16.428249    4962 scope.go:122] "RemoveContainer" containerID="ca202cfcda423dd3e6c21f34129a1ba08b8259feef77f870b6770e93107b154c"
Oct 29 22:00:16.428724 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:00:16.428696    4962 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8_odh-ai-gateway-infra(e6fcbf03-5017-48ca-a294-7b837a9d8877)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8" podUID="e6fcbf03-5017-48ca-a294-7b837a9d8877"
Oct 29 22:00:16.430378 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:16.430352    4962 generic.go:356] "Generic (PLEG): container finished" podID="c8765f54-99e8-4498-9591-2a7a1ca0059e" containerID="15191e773340df7f958ca999cf9d3e8b21979511e10c52ce9cdf85a87e26be8f" exitCode=60
Oct 29 22:00:16.430469 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:16.430431    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d" event={"ID":"c8765f54-99e8-4498-9591-2a7a1ca0059e","Type":"ContainerDied","Data":"15191e773340df7f958ca999cf9d3e8b21979511e10c52ce9cdf85a87e26be8f"}
Oct 29 22:00:16.430807 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:16.430791    4962 scope.go:122] "RemoveContainer" containerID="15191e773340df7f958ca999cf9d3e8b21979511e10c52ce9cdf85a87e26be8f"
Oct 29 22:00:16.431171 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:00:16.431150    4962 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d_odh-ai-gateway-infra(c8765f54-99e8-4498-9591-2a7a1ca0059e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d" podUID="c8765f54-99e8-4498-9591-2a7a1ca0059e"
Oct 29 22:00:16.509472 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:16.509445    4962 scope.go:122] "RemoveContainer" containerID="398d593824e39e5c2433b9b314be3726331d0f527cfec92d2726ee9f49e19c36"
Oct 29 22:00:17.468177 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:17.468147    4962 scope.go:122] "RemoveContainer" containerID="15191e773340df7f958ca999cf9d3e8b21979511e10c52ce9cdf85a87e26be8f"
Oct 29 22:00:17.468720 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:00:17.468557    4962 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d_odh-ai-gateway-infra(c8765f54-99e8-4498-9591-2a7a1ca0059e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d" podUID="c8765f54-99e8-4498-9591-2a7a1ca0059e"
Oct 29 22:00:17.473591 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:17.472574    4962 scope.go:122] "RemoveContainer" containerID="ca202cfcda423dd3e6c21f34129a1ba08b8259feef77f870b6770e93107b154c"
Oct 29 22:00:17.473591 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:00:17.472973    4962 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8_odh-ai-gateway-infra(e6fcbf03-5017-48ca-a294-7b837a9d8877)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8" podUID="e6fcbf03-5017-48ca-a294-7b837a9d8877"
Oct 29 22:00:19.796842 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:19.796808    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"]
Oct 29 22:00:19.827308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:19.802408    4962 cpu_manager.go:414] "containerMap: removing container" podUID="58be6b18-3bda-47b5-8c32-42e0b2e87fc1" containerName="main"
Oct 29 22:00:19.827527 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:19.827505    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58be6b18-3bda-47b5-8c32-42e0b2e87fc1" containerName="main"
Oct 29 22:00:19.852658 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:19.827865    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="58be6b18-3bda-47b5-8c32-42e0b2e87fc1" containerName="main"
Oct 29 22:00:19.954443 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:19.954396    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:19.958671 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:19.958646    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"]
Oct 29 22:00:19.958998 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:19.958982    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.042316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.040347    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.042316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.040459    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.042316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.040538    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.042316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.040564    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.042316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.040599    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8gx9\" (UniqueName: \"kubernetes.io/projected/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-kube-api-access-q8gx9\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.042316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.040680    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.042316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.040715    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.042316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.040791    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.042316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.040871    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.153679 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.152641    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.153679 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.152740    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.153679 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.152765    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.153679 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.152801    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8gx9\" (UniqueName: \"kubernetes.io/projected/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-kube-api-access-q8gx9\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.153679 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.152884    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.153679 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.152926    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.153679 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.153006    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.153679 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.153102    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.153679 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.153211    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.154238 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.153811    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-workload-certs\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.190335 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.176391    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:00:20.190335 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.178728    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-data\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.190335 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.185364    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-podinfo\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.190335 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.185683    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-workload-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.190335 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.185927    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-credential-socket\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.190335 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.186844    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istiod-ca-cert\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.212654 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.212595    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:20.241682 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.230189    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:20.263337 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.250235    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-envoy\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.264595 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.264561    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:00:20.324237 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.324194    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8gx9\" (UniqueName: \"kubernetes.io/projected/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-kube-api-access-q8gx9\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.431389 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.424594    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-token\") pod \"e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") " pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:20.576482 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.576429    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:00:20.612327 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.589737    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:20.696178 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:20.696088    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:21.006164 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:21.006065    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:21.024339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:21.007264    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:00:21.026009 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:21.019587    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:21.277378 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:21.213669    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:21.277378 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:21.241773    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:21.277378 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:21.242840    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:21.327308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:21.283394    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:21.327308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:21.285411    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:00:21.327308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:21.285484    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr" podUID="d91917d2-1f66-4ee4-a312-57398651739e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:21.593378 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:21.592340    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:00:22.035357 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:22.017100    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:22.060310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:22.027761    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded"
Oct 29 22:00:22.060310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:22.027842    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:22.498615 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:22.498553    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:22.530165 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:22.499634    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:22.566770 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:22.566581    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:00:22.705367 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:00:22.675170    4962 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.622s"
Oct 29 22:00:22.705367 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:22.675391    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 22:00:22.705367 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:22.675459    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 22:00:22.705367 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:22.675517    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 22:00:22.705367 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:22.675698    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 22:00:22.705367 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:22.675752    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:00:22.705367 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:22.675804    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 22:00:22.741692 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:22.741644    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:00:22.842310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:22.741904    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:22.860973 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:22.742163    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded"
Oct 29 22:00:23.069375 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:23.065432    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:00:23.069375 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:23.065960    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:00:23.517968 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:23.503232    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:23.557426 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:23.557374    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:23.609366 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:23.569462    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:23.739333 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:23.738170    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:00:23.739333 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:23.738392    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:00:23.739333 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:23.738485    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:00:23.870541 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:23.870434    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:00:23.885391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:23.835218    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:24.208337 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:24.208270    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:24.322932 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:24.322881    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:24.353417 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:24.353371    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded"
Oct 29 22:00:24.410313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:24.408311    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"]
Oct 29 22:00:24.414119 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:24.414089    4962 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-jdps8 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.17:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:00:24.414322 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:24.414297    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8" podUID="f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.17:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:24.553367 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:24.545477    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:00:24.610416 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:24.609854    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:24.612051 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:24.612019    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:24.876949 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:24.876431    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:00:24.876949 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:24.876498    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh" podUID="f06b758b-d09f-4882-a198-d88e91c7f390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:24.876949 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:24.876593    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:24.990738 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:24.990694    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:00:25.388136 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:25.345533    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:25.518310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:25.517680    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:00:25.587156 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:25.584575    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:25.638248 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:25.619543    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:25.674314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:25.652069    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:25.757311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:25.732785    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:00:25.757311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:25.732862    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr" podUID="4ef87fe2-429a-4393-8ddf-fa43e2dd9044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:25.757311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:25.733086    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:25.992524 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:25.992480    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:25.993222 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:25.993176    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:00:26.404370 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:26.404326    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:26.408269 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:26.408237    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf" event={"ID":"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8","Type":"ContainerStarted","Data":"2d6dad9233bd211c3c0e6e6725212a930fb91ec67a96ecc6e207ebdc439aaa89"}
Oct 29 22:00:26.518458 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:26.518364    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:26.589462 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:26.588370    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:00:26.626501 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:26.626465    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:00:26.626829 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:26.626786    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5" podUID="9762d8d0-a759-42ac-bc1b-c948adfa6f9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:26.648120 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:26.648076    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:26.676436 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:26.676392    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:26.995413 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:26.995368    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:26.996003 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:26.995970    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:00:27.053142 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:27.053105    4962 scope.go:122] "RemoveContainer" containerID="ca202cfcda423dd3e6c21f34129a1ba08b8259feef77f870b6770e93107b154c"
Oct 29 22:00:27.089391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:27.085829    4962 scope.go:122] "RemoveContainer" containerID="15191e773340df7f958ca999cf9d3e8b21979511e10c52ce9cdf85a87e26be8f"
Oct 29 22:00:27.090265 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:27.090238    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:00:27.090521 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:27.090472    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw" podUID="f6631eff-5d71-483c-aac7-8d578274ce76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:27.405622 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:27.405583    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:00:27.538870 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:27.531388    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:00:27.603375 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:27.593479    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:00:27.777046 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:27.652396    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:27.777046 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:27.709860    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:00:27.888179 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:27.887834    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded"
Oct 29 22:00:27.999466 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:27.999415    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:00:28.071033 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:28.070991    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:00:28.194181 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:28.194143    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf" event={"ID":"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8","Type":"ContainerStarted","Data":"b8938ecc7272a4a12de802fbe8b51101e19c7cf48dc91072654e0d3b6a0642ca"}
Oct 29 22:00:28.289309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:28.270000    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf" podStartSLOduration=9.269981353 podStartE2EDuration="9.269981353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:28.269409803 +0000 UTC m=+849.342366789" watchObservedRunningTime="2026-10-29 22:00:28.269981353 +0000 UTC m=+849.342938316"
Oct 29 22:00:28.456072 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:28.456032    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:28.689310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:28.646714    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:28.711860 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:28.710966    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:00:28.713331 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:28.712957    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:00:28.731312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:28.714407    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:00:28.789229 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:28.789193    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:28.938310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:28.930834    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:00:28.938609 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:28.938573    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf" podUID="84e0c5a1-5acf-4a80-8448-5aec9f52b0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 29 22:00:29.034691 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.034660    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"]
Oct 29 22:00:29.035546 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.035495    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:00:29.113861 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.113810    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:00:29.218983 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.218898    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:00:29.227071 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.227049    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 22:00:29.300925 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.297170    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-743a5e\"/\"distinct-2-w5-743a5e-kserve-self-signed-certs\""
Oct 29 22:00:29.325670 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.310629    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-dshm\") pod \"distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf\" (UID: \"ce61403b-7eab-4ba2-94e0-982fcea9e829\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:00:29.325670 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.310851    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-tmp-dir\") pod \"distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf\" (UID: \"ce61403b-7eab-4ba2-94e0-982fcea9e829\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:00:29.325670 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.310927    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-home\") pod \"distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf\" (UID: \"ce61403b-7eab-4ba2-94e0-982fcea9e829\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:00:29.325670 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.310986    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-model-cache\") pod \"distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf\" (UID: \"ce61403b-7eab-4ba2-94e0-982fcea9e829\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:00:29.325670 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.311063    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4989\" (UniqueName: \"kubernetes.io/projected/ce61403b-7eab-4ba2-94e0-982fcea9e829-kube-api-access-f4989\") pod \"distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf\" (UID: \"ce61403b-7eab-4ba2-94e0-982fcea9e829\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:00:29.325670 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.311102    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce61403b-7eab-4ba2-94e0-982fcea9e829-tls-certs\") pod \"distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf\" (UID: \"ce61403b-7eab-4ba2-94e0-982fcea9e829\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:00:29.451978 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.436071    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-tmp-dir\") pod \"distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf\" (UID: \"ce61403b-7eab-4ba2-94e0-982fcea9e829\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:00:29.452716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.452681    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-home\") pod \"distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf\" (UID: \"ce61403b-7eab-4ba2-94e0-982fcea9e829\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:00:29.453005 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.452985    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-model-cache\") pod \"distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf\" (UID: \"ce61403b-7eab-4ba2-94e0-982fcea9e829\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:00:29.453306 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.453273    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4989\" (UniqueName: \"kubernetes.io/projected/ce61403b-7eab-4ba2-94e0-982fcea9e829-kube-api-access-f4989\") pod \"distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf\" (UID: \"ce61403b-7eab-4ba2-94e0-982fcea9e829\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:00:29.453707 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.453689    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce61403b-7eab-4ba2-94e0-982fcea9e829-tls-certs\") pod \"distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf\" (UID: \"ce61403b-7eab-4ba2-94e0-982fcea9e829\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:00:29.453985 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.453965    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-dshm\") pod \"distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf\" (UID: \"ce61403b-7eab-4ba2-94e0-982fcea9e829\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:00:29.469662 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.469565    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-model-cache\") pod \"distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf\" (UID: \"ce61403b-7eab-4ba2-94e0-982fcea9e829\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:00:29.470166 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.470124    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-tmp-dir\") pod \"distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf\" (UID: \"ce61403b-7eab-4ba2-94e0-982fcea9e829\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:00:29.470531 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.470494    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-home\") pod \"distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf\" (UID: \"ce61403b-7eab-4ba2-94e0-982fcea9e829\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:00:29.515462 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.513795    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce61403b-7eab-4ba2-94e0-982fcea9e829-tls-certs\") pod \"distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf\" (UID: \"ce61403b-7eab-4ba2-94e0-982fcea9e829\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:00:29.591375 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.589953    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4989\" (UniqueName: \"kubernetes.io/projected/ce61403b-7eab-4ba2-94e0-982fcea9e829-kube-api-access-f4989\") pod \"distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf\" (UID: \"ce61403b-7eab-4ba2-94e0-982fcea9e829\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:00:29.599715 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.599610    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-dshm\") pod \"distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf\" (UID: \"ce61403b-7eab-4ba2-94e0-982fcea9e829\") " pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:00:29.611315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.609972    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 22:00:29.611315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.610035    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:00:29.611315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.610057    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"]
Oct 29 22:00:29.611315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.610151    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 22:00:29.611315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.610194    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 22:00:29.611315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.610226    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:00:29.611315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.610257    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 22:00:29.721602 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.721517    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:00:29.721854 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.721814    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf" podUID="84e0c5a1-5acf-4a80-8448-5aec9f52b0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 29 22:00:29.805811 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:29.804587    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:00:30.228309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:30.225009    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"]
Oct 29 22:00:30.627805 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:30.627725    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" event={"ID":"ce61403b-7eab-4ba2-94e0-982fcea9e829","Type":"ContainerStarted","Data":"3591c3f23ae7dd91fc821238a9cc5895d79ca54231c801125ff9b0c5d7697e4c"}
Oct 29 22:00:30.627805 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:30.627770    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" event={"ID":"ce61403b-7eab-4ba2-94e0-982fcea9e829","Type":"ContainerStarted","Data":"74614c14e908aca3fe4bda53783672f3c4b058519dfed52eb9dbaa12d18340f7"}
Oct 29 22:00:30.629713 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:30.629688    4962 generic.go:356] "Generic (PLEG): container finished" podID="c8765f54-99e8-4498-9591-2a7a1ca0059e" containerID="41eeca110855968dfd94bb037cde2b1ba0852f5b3e5c370c0b668bbb88e05bb4" exitCode=60
Oct 29 22:00:30.629811 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:30.629760    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d" event={"ID":"c8765f54-99e8-4498-9591-2a7a1ca0059e","Type":"ContainerDied","Data":"41eeca110855968dfd94bb037cde2b1ba0852f5b3e5c370c0b668bbb88e05bb4"}
Oct 29 22:00:30.629878 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:30.629812    4962 scope.go:122] "RemoveContainer" containerID="15191e773340df7f958ca999cf9d3e8b21979511e10c52ce9cdf85a87e26be8f"
Oct 29 22:00:30.630188 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:30.630169    4962 scope.go:122] "RemoveContainer" containerID="41eeca110855968dfd94bb037cde2b1ba0852f5b3e5c370c0b668bbb88e05bb4"
Oct 29 22:00:30.630589 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:00:30.630569    4962 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d_odh-ai-gateway-infra(c8765f54-99e8-4498-9591-2a7a1ca0059e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d" podUID="c8765f54-99e8-4498-9591-2a7a1ca0059e"
Oct 29 22:00:30.631449 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:30.631429    4962 generic.go:356] "Generic (PLEG): container finished" podID="e6fcbf03-5017-48ca-a294-7b837a9d8877" containerID="200e77dc78022ee3a3ff9630b835a1c4a01f907258bb504c900e4ed157312dfc" exitCode=60
Oct 29 22:00:30.631727 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:30.631708    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8" event={"ID":"e6fcbf03-5017-48ca-a294-7b837a9d8877","Type":"ContainerDied","Data":"200e77dc78022ee3a3ff9630b835a1c4a01f907258bb504c900e4ed157312dfc"}
Oct 29 22:00:30.632057 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:30.632044    4962 scope.go:122] "RemoveContainer" containerID="200e77dc78022ee3a3ff9630b835a1c4a01f907258bb504c900e4ed157312dfc"
Oct 29 22:00:30.632332 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:00:30.632282    4962 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8_odh-ai-gateway-infra(e6fcbf03-5017-48ca-a294-7b837a9d8877)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8" podUID="e6fcbf03-5017-48ca-a294-7b837a9d8877"
Oct 29 22:00:30.642993 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:30.642975    4962 scope.go:122] "RemoveContainer" containerID="ca202cfcda423dd3e6c21f34129a1ba08b8259feef77f870b6770e93107b154c"
Oct 29 22:00:30.695318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:30.692653    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podStartSLOduration=2.69263354 podStartE2EDuration="2.69263354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:30.669154676 +0000 UTC m=+851.742111653" watchObservedRunningTime="2026-10-29 22:00:30.69263354 +0000 UTC m=+851.765590506"
Oct 29 22:00:30.721845 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:30.721820    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:00:30.721970 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:30.721870    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf" podUID="84e0c5a1-5acf-4a80-8448-5aec9f52b0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 29 22:00:30.819334 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:30.819055    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:00:31.721588 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:31.721555    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:00:31.721990 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:31.721611    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf" podUID="84e0c5a1-5acf-4a80-8448-5aec9f52b0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 29 22:00:31.735214 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:31.735188    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8"]
Oct 29 22:00:31.802972 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:31.802345    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d"]
Oct 29 22:00:32.063239 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.061500    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8"
Oct 29 22:00:32.164118 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.164049    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6fcbf03-5017-48ca-a294-7b837a9d8877-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6fcbf03-5017-48ca-a294-7b837a9d8877-maas-api-service-ca\") pod \"e6fcbf03-5017-48ca-a294-7b837a9d8877\" (UID: \"e6fcbf03-5017-48ca-a294-7b837a9d8877\") "
Oct 29 22:00:32.167303 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.164304    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6fcbf03-5017-48ca-a294-7b837a9d8877-kube-api-access-6ttxf\" (UniqueName: \"kubernetes.io/projected/e6fcbf03-5017-48ca-a294-7b837a9d8877-kube-api-access-6ttxf\") pod \"e6fcbf03-5017-48ca-a294-7b837a9d8877\" (UID: \"e6fcbf03-5017-48ca-a294-7b837a9d8877\") "
Oct 29 22:00:32.167303 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.164464    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6fcbf03-5017-48ca-a294-7b837a9d8877-maas-api-service-ca" pod "e6fcbf03-5017-48ca-a294-7b837a9d8877" (UID: "e6fcbf03-5017-48ca-a294-7b837a9d8877"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:00:32.167303 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.164977    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6fcbf03-5017-48ca-a294-7b837a9d8877-maas-api-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:00:32.167303 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.166632    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fcbf03-5017-48ca-a294-7b837a9d8877-kube-api-access-6ttxf" pod "e6fcbf03-5017-48ca-a294-7b837a9d8877" (UID: "e6fcbf03-5017-48ca-a294-7b837a9d8877"). InnerVolumeSpecName "kube-api-access-6ttxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:00:32.266976 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.266947    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ttxf\" (UniqueName: \"kubernetes.io/projected/e6fcbf03-5017-48ca-a294-7b837a9d8877-kube-api-access-6ttxf\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:00:32.300309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.286780    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d"
Oct 29 22:00:32.367922 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.367885    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8765f54-99e8-4498-9591-2a7a1ca0059e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8765f54-99e8-4498-9591-2a7a1ca0059e-maas-api-service-ca\") pod \"c8765f54-99e8-4498-9591-2a7a1ca0059e\" (UID: \"c8765f54-99e8-4498-9591-2a7a1ca0059e\") "
Oct 29 22:00:32.368106 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.368088    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8765f54-99e8-4498-9591-2a7a1ca0059e-kube-api-access-f5kxm\" (UniqueName: \"kubernetes.io/projected/c8765f54-99e8-4498-9591-2a7a1ca0059e-kube-api-access-f5kxm\") pod \"c8765f54-99e8-4498-9591-2a7a1ca0059e\" (UID: \"c8765f54-99e8-4498-9591-2a7a1ca0059e\") "
Oct 29 22:00:32.368274 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.368243    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8765f54-99e8-4498-9591-2a7a1ca0059e-maas-api-service-ca" pod "c8765f54-99e8-4498-9591-2a7a1ca0059e" (UID: "c8765f54-99e8-4498-9591-2a7a1ca0059e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:00:32.368778 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.368759    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8765f54-99e8-4498-9591-2a7a1ca0059e-maas-api-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:00:32.370278 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.370254    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8765f54-99e8-4498-9591-2a7a1ca0059e-kube-api-access-f5kxm" pod "c8765f54-99e8-4498-9591-2a7a1ca0059e" (UID: "c8765f54-99e8-4498-9591-2a7a1ca0059e"). InnerVolumeSpecName "kube-api-access-f5kxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:00:32.470513 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.470483    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5kxm\" (UniqueName: \"kubernetes.io/projected/c8765f54-99e8-4498-9591-2a7a1ca0059e-kube-api-access-f5kxm\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:00:32.650957 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.650873    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d"
Oct 29 22:00:32.650957 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.650896    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d" event={"ID":"c8765f54-99e8-4498-9591-2a7a1ca0059e","Type":"ContainerDied","Data":"93f3fbcfd4af33fae0186f7eb4aa7f335cb95b57000287bd9fb0e1e37e17ba24"}
Oct 29 22:00:32.651247 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.650957    4962 scope.go:122] "RemoveContainer" containerID="41eeca110855968dfd94bb037cde2b1ba0852f5b3e5c370c0b668bbb88e05bb4"
Oct 29 22:00:32.652446 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.652428    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8"
Oct 29 22:00:32.652557 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.652479    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8" event={"ID":"e6fcbf03-5017-48ca-a294-7b837a9d8877","Type":"ContainerDied","Data":"7727b625b111311a7de8c1d102a4dce0655626fe9bf9238a7a73d563438551ed"}
Oct 29 22:00:32.662855 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.662833    4962 scope.go:122] "RemoveContainer" containerID="200e77dc78022ee3a3ff9630b835a1c4a01f907258bb504c900e4ed157312dfc"
Oct 29 22:00:32.716200 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.716173    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8"]
Oct 29 22:00:32.721183 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.721153    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:00:32.721317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.721199    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf" podUID="84e0c5a1-5acf-4a80-8448-5aec9f52b0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 29 22:00:32.722632 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.722613    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a940bc-29888520-nm6b8"]
Oct 29 22:00:32.758914 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.758892    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d"]
Oct 29 22:00:32.765333 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:32.765313    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-743a5e-29888520-2fg6d"]
Oct 29 22:00:33.060261 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:33.060231    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8765f54-99e8-4498-9591-2a7a1ca0059e" path="/var/lib/kubelet/pods/c8765f54-99e8-4498-9591-2a7a1ca0059e/volumes"
Oct 29 22:00:33.060682 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:33.060668    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6fcbf03-5017-48ca-a294-7b837a9d8877" path="/var/lib/kubelet/pods/e6fcbf03-5017-48ca-a294-7b837a9d8877/volumes"
Oct 29 22:00:33.729086 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:33.729036    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:00:33.729800 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:33.729765    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf" podUID="84e0c5a1-5acf-4a80-8448-5aec9f52b0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 29 22:00:34.721324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:34.721276    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:00:34.721543 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:34.721361    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf" podUID="84e0c5a1-5acf-4a80-8448-5aec9f52b0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 29 22:00:35.731311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:35.730400    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:35.732919 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:35.732809    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 29 22:00:35.732919 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:35.732878    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf" podUID="84e0c5a1-5acf-4a80-8448-5aec9f52b0b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 29 22:00:36.725304 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:36.725257    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:36.726400 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:36.726371    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:00:39.767482 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:00:39.767440    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:00:39.805263 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:39.805233    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:00:39.818440 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:39.818418    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:00:39.985533 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:39.985502    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:00:40.188957 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.188927    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"]
Oct 29 22:00:40.189659 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.189639    4962 cpu_manager.go:414] "containerMap: removing container" podUID="c8765f54-99e8-4498-9591-2a7a1ca0059e" containerName="cleanup"
Oct 29 22:00:40.189659 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.189659    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8765f54-99e8-4498-9591-2a7a1ca0059e" containerName="cleanup"
Oct 29 22:00:40.189800 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.189669    4962 cpu_manager.go:414] "containerMap: removing container" podUID="e6fcbf03-5017-48ca-a294-7b837a9d8877" containerName="cleanup"
Oct 29 22:00:40.189800 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.189675    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6fcbf03-5017-48ca-a294-7b837a9d8877" containerName="cleanup"
Oct 29 22:00:40.189800 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.189699    4962 cpu_manager.go:414] "containerMap: removing container" podUID="c8765f54-99e8-4498-9591-2a7a1ca0059e" containerName="cleanup"
Oct 29 22:00:40.189800 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.189704    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8765f54-99e8-4498-9591-2a7a1ca0059e" containerName="cleanup"
Oct 29 22:00:40.189800 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.189710    4962 cpu_manager.go:414] "containerMap: removing container" podUID="e6fcbf03-5017-48ca-a294-7b837a9d8877" containerName="cleanup"
Oct 29 22:00:40.189800 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.189714    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6fcbf03-5017-48ca-a294-7b837a9d8877" containerName="cleanup"
Oct 29 22:00:40.189800 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.189725    4962 cpu_manager.go:414] "containerMap: removing container" podUID="e6fcbf03-5017-48ca-a294-7b837a9d8877" containerName="cleanup"
Oct 29 22:00:40.189800 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.189730    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6fcbf03-5017-48ca-a294-7b837a9d8877" containerName="cleanup"
Oct 29 22:00:40.190029 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.189881    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8765f54-99e8-4498-9591-2a7a1ca0059e" containerName="cleanup"
Oct 29 22:00:40.190029 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.189890    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6fcbf03-5017-48ca-a294-7b837a9d8877" containerName="cleanup"
Oct 29 22:00:40.190029 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.189907    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6fcbf03-5017-48ca-a294-7b837a9d8877" containerName="cleanup"
Oct 29 22:00:40.190029 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.189916    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8765f54-99e8-4498-9591-2a7a1ca0059e" containerName="cleanup"
Oct 29 22:00:40.193670 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.193648    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.220695 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.220668    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"]
Oct 29 22:00:40.261126 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.261091    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.261299 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.261169    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.261299 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.261198    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1897161-0016-400a-9928-8b407f4148e9-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.261436 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.261327    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1897161-0016-400a-9928-8b407f4148e9-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.261436 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.261406    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.261554 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.261471    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1897161-0016-400a-9928-8b407f4148e9-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.261554 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.261504    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzhzx\" (UniqueName: \"kubernetes.io/projected/e1897161-0016-400a-9928-8b407f4148e9-kube-api-access-jzhzx\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.261554 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.261524    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.261712 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.261618    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.362654 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.362624    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.362787 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.362703    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1897161-0016-400a-9928-8b407f4148e9-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.362787 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.362738    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzhzx\" (UniqueName: \"kubernetes.io/projected/e1897161-0016-400a-9928-8b407f4148e9-kube-api-access-jzhzx\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.362787 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.362762    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.362912 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.362798    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.362975 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.362930    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.363043 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.362999    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.363043 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.363026    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1897161-0016-400a-9928-8b407f4148e9-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.363160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.363097    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1897161-0016-400a-9928-8b407f4148e9-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.363160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.363123    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-workload-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.363267 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.363186    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-credential-socket\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.363346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.363323    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-workload-certs\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.363655 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.363636    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-istio-data\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.363892 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.363871    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1897161-0016-400a-9928-8b407f4148e9-istiod-ca-cert\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.364934 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.364899    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-istio-envoy\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.365244 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.365226    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1897161-0016-400a-9928-8b407f4148e9-istio-podinfo\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.386222 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.386191    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1897161-0016-400a-9928-8b407f4148e9-istio-token\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.386470 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.386453    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzhzx\" (UniqueName: \"kubernetes.io/projected/e1897161-0016-400a-9928-8b407f4148e9-kube-api-access-jzhzx\") pod \"e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") " pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:40.541358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:40.514920    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:41.037462 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:41.037439    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"]
Oct 29 22:00:41.039365 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:00:41.039337    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1897161_0016_400a_9928_8b407f4148e9.slice/crio-4b653a55a89d52585ac1053c22a09803c1866666694976ef947d3918776881b9 WatchSource:0}: Error finding container 4b653a55a89d52585ac1053c22a09803c1866666694976ef947d3918776881b9: Status 404 returned error can't find the container with id 4b653a55a89d52585ac1053c22a09803c1866666694976ef947d3918776881b9
Oct 29 22:00:41.041768 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:41.041739    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:00:41.041843 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:41.041800    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:00:41.041843 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:41.041829    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:00:42.004369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:42.004330    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42" event={"ID":"e1897161-0016-400a-9928-8b407f4148e9","Type":"ContainerStarted","Data":"2fb3d9576ee00f91c232b0db8a4d81c05a44ba966996f0907fb75967ce88eb72"}
Oct 29 22:00:42.004614 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:42.004390    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42" event={"ID":"e1897161-0016-400a-9928-8b407f4148e9","Type":"ContainerStarted","Data":"4b653a55a89d52585ac1053c22a09803c1866666694976ef947d3918776881b9"}
Oct 29 22:00:42.063623 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:42.063570    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42" podStartSLOduration=2.063556849 podStartE2EDuration="2.063556849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:42.061839567 +0000 UTC m=+863.134796569" watchObservedRunningTime="2026-10-29 22:00:42.063556849 +0000 UTC m=+863.136513877"
Oct 29 22:00:42.531479 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:42.529980    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:42.541062 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:42.538433    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:00:42.541062 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:42.538503    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42" podUID="e1897161-0016-400a-9928-8b407f4148e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 29 22:00:43.528090 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:43.528056    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:00:43.528522 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:43.528111    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42" podUID="e1897161-0016-400a-9928-8b407f4148e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 29 22:00:44.532953 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:44.532916    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:00:44.533817 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:44.533776    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42" podUID="e1897161-0016-400a-9928-8b407f4148e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 29 22:00:45.527962 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:45.527919    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:00:45.528132 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:45.527985    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42" podUID="e1897161-0016-400a-9928-8b407f4148e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 29 22:00:46.528120 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:46.528086    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:00:46.528532 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:46.528139    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42" podUID="e1897161-0016-400a-9928-8b407f4148e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 29 22:00:47.528602 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:47.528563    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:00:47.529005 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:47.528640    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42" podUID="e1897161-0016-400a-9928-8b407f4148e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 29 22:00:48.531933 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:48.531901    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:48.532415 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:48.532392    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:48.533116 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:48.533097    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:00:50.600188 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.600149    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"]
Oct 29 22:00:50.608514 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.608484    4962 cpu_manager.go:414] "containerMap: removing container" podUID="c8765f54-99e8-4498-9591-2a7a1ca0059e" containerName="cleanup"
Oct 29 22:00:50.608746 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.608727    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8765f54-99e8-4498-9591-2a7a1ca0059e" containerName="cleanup"
Oct 29 22:00:50.609137 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.609118    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6fcbf03-5017-48ca-a294-7b837a9d8877" containerName="cleanup"
Oct 29 22:00:50.609262 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.609253    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8765f54-99e8-4498-9591-2a7a1ca0059e" containerName="cleanup"
Oct 29 22:00:50.622708 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.622679    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.637589 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.637509    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"]
Oct 29 22:00:50.735042 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.735002    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.735206 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.735148    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.735206 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.735200    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.735353 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.735328    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64a76b6c-1542-4fb1-9730-7b770c452b56-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.735408 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.735382    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.735483 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.735467    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.735541 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.735503    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpwj9\" (UniqueName: \"kubernetes.io/projected/64a76b6c-1542-4fb1-9730-7b770c452b56-kube-api-access-xpwj9\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.735588 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.735548    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.735588 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.735581    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.836895 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.836863    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.836895 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.836899    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.837083 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.836926    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpwj9\" (UniqueName: \"kubernetes.io/projected/64a76b6c-1542-4fb1-9730-7b770c452b56-kube-api-access-xpwj9\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.837083 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.836958    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.837083 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.836975    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.837313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.837098    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.837313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.837209    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.837313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.837260    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.837492 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.837337    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64a76b6c-1542-4fb1-9730-7b770c452b56-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.837492 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.837417    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-workload-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.837492 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.837442    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-credential-socket\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.837660 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.837572    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-workload-certs\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.837660 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.837622    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-data\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.838148 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.838125    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64a76b6c-1542-4fb1-9730-7b770c452b56-istiod-ca-cert\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.839474 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.839450    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-envoy\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.839633 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.839612    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-podinfo\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.860562 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.860499    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-token\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.860671 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.860564    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpwj9\" (UniqueName: \"kubernetes.io/projected/64a76b6c-1542-4fb1-9730-7b770c452b56-kube-api-access-xpwj9\") pod \"e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") " pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:50.949958 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:50.949936    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:51.304458 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:00:51.304424    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a76b6c_1542_4fb1_9730_7b770c452b56.slice/crio-c03d6a48ec0b199a94d95c42c0d8845213a8036fda1eef4939a0b36e111d3a35 WatchSource:0}: Error finding container c03d6a48ec0b199a94d95c42c0d8845213a8036fda1eef4939a0b36e111d3a35: Status 404 returned error can't find the container with id c03d6a48ec0b199a94d95c42c0d8845213a8036fda1eef4939a0b36e111d3a35
Oct 29 22:00:51.306766 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:51.306724    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"]
Oct 29 22:00:51.307720 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:51.307691    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:00:51.307785 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:51.307750    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:00:51.307785 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:51.307777    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:00:52.108793 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:52.108749    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6" event={"ID":"64a76b6c-1542-4fb1-9730-7b770c452b56","Type":"ContainerStarted","Data":"fc31966e8412d3b25aade1df656c1c99dff20fed6545634e537352070118b61c"}
Oct 29 22:00:52.109226 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:52.108811    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6" event={"ID":"64a76b6c-1542-4fb1-9730-7b770c452b56","Type":"ContainerStarted","Data":"c03d6a48ec0b199a94d95c42c0d8845213a8036fda1eef4939a0b36e111d3a35"}
Oct 29 22:00:52.169610 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:52.169563    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6" podStartSLOduration=2.169549586 podStartE2EDuration="2.169549586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:00:52.165788831 +0000 UTC m=+873.238745794" watchObservedRunningTime="2026-10-29 22:00:52.169549586 +0000 UTC m=+873.242506550"
Oct 29 22:00:52.956325 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:52.953639    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:52.958092 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:52.958057    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:00:52.958352 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:52.958323    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6" podUID="64a76b6c-1542-4fb1-9730-7b770c452b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 29 22:00:53.950244 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:53.950208    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:00:53.950648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:53.950273    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6" podUID="64a76b6c-1542-4fb1-9730-7b770c452b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 29 22:00:54.950733 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:54.950702    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:00:54.951151 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:54.950768    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6" podUID="64a76b6c-1542-4fb1-9730-7b770c452b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 29 22:00:55.950592 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:55.950558    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:00:55.950786 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:55.950619    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6" podUID="64a76b6c-1542-4fb1-9730-7b770c452b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 29 22:00:56.950756 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:56.950719    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:00:56.950927 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:56.950778    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6" podUID="64a76b6c-1542-4fb1-9730-7b770c452b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 29 22:00:57.950578 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:57.950543    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 29 22:00:57.950752 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:57.950610    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6" podUID="64a76b6c-1542-4fb1-9730-7b770c452b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 29 22:00:58.954845 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:58.954815    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:58.955344 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:58.955134    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:00:58.955821 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:00:58.955797    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:01:09.013714 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.013679    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"]
Oct 29 22:01:09.020024 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.019998    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:09.041984 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.041952    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"]
Oct 29 22:01:09.047307 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.047272    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a3c02e\"/\"kube-root-ca.crt\""
Oct 29 22:01:09.047463 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.047331    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a3c02e\"/\"openshift-service-ca.crt\""
Oct 29 22:01:09.047463 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.047434    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a3c02e\"/\"distinct-2-w1-a3c02e-kserve-self-signed-certs\""
Oct 29 22:01:09.047622 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.047465    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a3c02e\"/\"default-dockercfg-5krsk\""
Oct 29 22:01:09.149118 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.149087    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbcad86b-328d-49a0-a993-63dc68f154a4-tls-certs\") pod \"distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk\" (UID: \"fbcad86b-328d-49a0-a993-63dc68f154a4\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:09.149280 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.149143    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-tmp-dir\") pod \"distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk\" (UID: \"fbcad86b-328d-49a0-a993-63dc68f154a4\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:09.149280 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.149216    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t56t8\" (UniqueName: \"kubernetes.io/projected/fbcad86b-328d-49a0-a993-63dc68f154a4-kube-api-access-t56t8\") pod \"distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk\" (UID: \"fbcad86b-328d-49a0-a993-63dc68f154a4\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:09.149280 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.149243    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-dshm\") pod \"distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk\" (UID: \"fbcad86b-328d-49a0-a993-63dc68f154a4\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:09.149280 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.149262    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-home\") pod \"distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk\" (UID: \"fbcad86b-328d-49a0-a993-63dc68f154a4\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:09.149530 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.149506    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-model-cache\") pod \"distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk\" (UID: \"fbcad86b-328d-49a0-a993-63dc68f154a4\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:09.258439 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.258386    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-model-cache\") pod \"distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk\" (UID: \"fbcad86b-328d-49a0-a993-63dc68f154a4\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:09.259121 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.259090    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbcad86b-328d-49a0-a993-63dc68f154a4-tls-certs\") pod \"distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk\" (UID: \"fbcad86b-328d-49a0-a993-63dc68f154a4\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:09.265584 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.265491    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-tmp-dir\") pod \"distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk\" (UID: \"fbcad86b-328d-49a0-a993-63dc68f154a4\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:09.265845 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.265806    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t56t8\" (UniqueName: \"kubernetes.io/projected/fbcad86b-328d-49a0-a993-63dc68f154a4-kube-api-access-t56t8\") pod \"distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk\" (UID: \"fbcad86b-328d-49a0-a993-63dc68f154a4\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:09.266048 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.266028    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-dshm\") pod \"distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk\" (UID: \"fbcad86b-328d-49a0-a993-63dc68f154a4\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:09.266197 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.266181    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-home\") pod \"distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk\" (UID: \"fbcad86b-328d-49a0-a993-63dc68f154a4\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:09.289311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.268482    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-tmp-dir\") pod \"distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk\" (UID: \"fbcad86b-328d-49a0-a993-63dc68f154a4\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:09.290196 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.290164    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-home\") pod \"distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk\" (UID: \"fbcad86b-328d-49a0-a993-63dc68f154a4\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:09.291148 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.291120    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-model-cache\") pod \"distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk\" (UID: \"fbcad86b-328d-49a0-a993-63dc68f154a4\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:09.330737 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.330658    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-dshm\") pod \"distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk\" (UID: \"fbcad86b-328d-49a0-a993-63dc68f154a4\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:09.371367 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.338267    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbcad86b-328d-49a0-a993-63dc68f154a4-tls-certs\") pod \"distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk\" (UID: \"fbcad86b-328d-49a0-a993-63dc68f154a4\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:09.372414 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.372268    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t56t8\" (UniqueName: \"kubernetes.io/projected/fbcad86b-328d-49a0-a993-63dc68f154a4-kube-api-access-t56t8\") pod \"distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk\" (UID: \"fbcad86b-328d-49a0-a993-63dc68f154a4\") " pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:09.639800 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.639722    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:09.738334 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:01:09.737456    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:01:09.953259 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:09.953228    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"]
Oct 29 22:01:09.953601 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:01:09.953573    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbcad86b_328d_49a0_a993_63dc68f154a4.slice/crio-4b2c60272b608331d3ff1552c9a479351970c5f699ec0a53f01a21ca7950d70d WatchSource:0}: Error finding container 4b2c60272b608331d3ff1552c9a479351970c5f699ec0a53f01a21ca7950d70d: Status 404 returned error can't find the container with id 4b2c60272b608331d3ff1552c9a479351970c5f699ec0a53f01a21ca7950d70d
Oct 29 22:01:10.300926 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:10.300834    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" event={"ID":"fbcad86b-328d-49a0-a993-63dc68f154a4","Type":"ContainerStarted","Data":"dc0df5cada975a571e11ca34856ddd616cb7649ab2f45b080b8806380b5e66c3"}
Oct 29 22:01:10.300926 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:10.300877    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" event={"ID":"fbcad86b-328d-49a0-a993-63dc68f154a4","Type":"ContainerStarted","Data":"4b2c60272b608331d3ff1552c9a479351970c5f699ec0a53f01a21ca7950d70d"}
Oct 29 22:01:10.346055 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:10.345997    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podStartSLOduration=2.345982775 podStartE2EDuration="2.345982775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:01:10.343869903 +0000 UTC m=+891.416826850" watchObservedRunningTime="2026-10-29 22:01:10.345982775 +0000 UTC m=+891.418939777"
Oct 29 22:01:10.641370 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:10.640464    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:14.097323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:14.097277    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"]
Oct 29 22:01:14.097765 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:14.097448    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk" podUID="029c7ea5-bf31-4b22-96b8-41a693287efe" containerName="istio-proxy" containerID="cri-o://e7db17f06ec1c34a7c7da47638cebdfa2c9c6da89800d405626cbb9c11101551" gracePeriod=30
Oct 29 22:01:14.111157 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:14.111129    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"]
Oct 29 22:01:14.111456 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:14.111340    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5" podUID="3dcff5f2-db63-4ae5-a935-7509c28c8867" containerName="istio-proxy" containerID="cri-o://dcb6295765a76eb116d2a684331e402fcd7bf11bcfaad16fe2829f4f0dbb605d" gracePeriod=30
Oct 29 22:01:19.512071 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.505727    4962 generic.go:356] "Generic (PLEG): container finished" podID="3dcff5f2-db63-4ae5-a935-7509c28c8867" containerID="dcb6295765a76eb116d2a684331e402fcd7bf11bcfaad16fe2829f4f0dbb605d" exitCode=0
Oct 29 22:01:19.512071 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.505819    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 22:01:19.512071 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.505821    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5" event={"ID":"3dcff5f2-db63-4ae5-a935-7509c28c8867","Type":"ContainerDied","Data":"dcb6295765a76eb116d2a684331e402fcd7bf11bcfaad16fe2829f4f0dbb605d"}
Oct 29 22:01:19.512071 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.511906    4962 generic.go:356] "Generic (PLEG): container finished" podID="029c7ea5-bf31-4b22-96b8-41a693287efe" containerID="e7db17f06ec1c34a7c7da47638cebdfa2c9c6da89800d405626cbb9c11101551" exitCode=0
Oct 29 22:01:19.512719 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.512146    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk" event={"ID":"029c7ea5-bf31-4b22-96b8-41a693287efe","Type":"ContainerDied","Data":"e7db17f06ec1c34a7c7da47638cebdfa2c9c6da89800d405626cbb9c11101551"}
Oct 29 22:01:19.512719 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.512188    4962 scope.go:122] "RemoveContainer" containerID="e7db17f06ec1c34a7c7da47638cebdfa2c9c6da89800d405626cbb9c11101551"
Oct 29 22:01:19.540915 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.540885    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/2.log"
Oct 29 22:01:19.542128 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.542107    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/2.log"
Oct 29 22:01:19.548350 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.548331    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74bdq_259e6b95-d64c-4101-871f-4a933d4ac38f/ovn-acl-logging/0.log"
Oct 29 22:01:19.549981 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.549963    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74bdq_259e6b95-d64c-4101-871f-4a933d4ac38f/ovn-acl-logging/0.log"
Oct 29 22:01:19.614384 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.614349    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/029c7ea5-bf31-4b22-96b8-41a693287efe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029c7ea5-bf31-4b22-96b8-41a693287efe-istiod-ca-cert\") pod \"029c7ea5-bf31-4b22-96b8-41a693287efe\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") "
Oct 29 22:01:19.614543 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.614429    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-workload-certs\") pod \"029c7ea5-bf31-4b22-96b8-41a693287efe\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") "
Oct 29 22:01:19.614543 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.614489    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-envoy\") pod \"029c7ea5-bf31-4b22-96b8-41a693287efe\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") "
Oct 29 22:01:19.614668 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.614636    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-credential-socket\") pod \"029c7ea5-bf31-4b22-96b8-41a693287efe\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") "
Oct 29 22:01:19.614733 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.614703    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-podinfo\") pod \"029c7ea5-bf31-4b22-96b8-41a693287efe\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") "
Oct 29 22:01:19.614841 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.614821    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-workload-socket\") pod \"029c7ea5-bf31-4b22-96b8-41a693287efe\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") "
Oct 29 22:01:19.614937 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.614921    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-token\" (UniqueName: \"kubernetes.io/projected/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-token\") pod \"029c7ea5-bf31-4b22-96b8-41a693287efe\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") "
Oct 29 22:01:19.615000 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.614974    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-data\") pod \"029c7ea5-bf31-4b22-96b8-41a693287efe\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") "
Oct 29 22:01:19.615069 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.615052    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/029c7ea5-bf31-4b22-96b8-41a693287efe-kube-api-access-r9r24\" (UniqueName: \"kubernetes.io/projected/029c7ea5-bf31-4b22-96b8-41a693287efe-kube-api-access-r9r24\") pod \"029c7ea5-bf31-4b22-96b8-41a693287efe\" (UID: \"029c7ea5-bf31-4b22-96b8-41a693287efe\") "
Oct 29 22:01:19.615748 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.614945    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/029c7ea5-bf31-4b22-96b8-41a693287efe-istiod-ca-cert" pod "029c7ea5-bf31-4b22-96b8-41a693287efe" (UID: "029c7ea5-bf31-4b22-96b8-41a693287efe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:01:19.615748 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.615626    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-workload-certs" pod "029c7ea5-bf31-4b22-96b8-41a693287efe" (UID: "029c7ea5-bf31-4b22-96b8-41a693287efe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:01:19.615748 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.615676    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-workload-socket" pod "029c7ea5-bf31-4b22-96b8-41a693287efe" (UID: "029c7ea5-bf31-4b22-96b8-41a693287efe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:01:19.615954 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.615821    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:01:19.615954 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.615842    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029c7ea5-bf31-4b22-96b8-41a693287efe-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:01:19.615954 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.615856    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:01:19.615954 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.615890    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-data" pod "029c7ea5-bf31-4b22-96b8-41a693287efe" (UID: "029c7ea5-bf31-4b22-96b8-41a693287efe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:01:19.616465 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.616398    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-credential-socket" pod "029c7ea5-bf31-4b22-96b8-41a693287efe" (UID: "029c7ea5-bf31-4b22-96b8-41a693287efe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:01:19.628773 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.627790    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029c7ea5-bf31-4b22-96b8-41a693287efe-kube-api-access-r9r24" pod "029c7ea5-bf31-4b22-96b8-41a693287efe" (UID: "029c7ea5-bf31-4b22-96b8-41a693287efe"). InnerVolumeSpecName "kube-api-access-r9r24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:01:19.628773 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.627948    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-envoy" pod "029c7ea5-bf31-4b22-96b8-41a693287efe" (UID: "029c7ea5-bf31-4b22-96b8-41a693287efe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:01:19.628773 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.628405    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-podinfo" pod "029c7ea5-bf31-4b22-96b8-41a693287efe" (UID: "029c7ea5-bf31-4b22-96b8-41a693287efe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:01:19.629972 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.629943    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-token" pod "029c7ea5-bf31-4b22-96b8-41a693287efe" (UID: "029c7ea5-bf31-4b22-96b8-41a693287efe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:01:19.640256 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.640236    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:19.653226 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.653202    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:19.720655 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.720236    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:01:19.720655 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.720268    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:01:19.720655 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.720282    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9r24\" (UniqueName: \"kubernetes.io/projected/029c7ea5-bf31-4b22-96b8-41a693287efe-kube-api-access-r9r24\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:01:19.720655 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.720313    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:01:19.720655 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.720325    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029c7ea5-bf31-4b22-96b8-41a693287efe-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:01:19.720655 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.720339    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029c7ea5-bf31-4b22-96b8-41a693287efe-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:01:19.792472 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.792449    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 22:01:19.922415 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.922382    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-workload-certs\") pod \"3dcff5f2-db63-4ae5-a935-7509c28c8867\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") "
Oct 29 22:01:19.922582 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.922430    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-workload-socket\") pod \"3dcff5f2-db63-4ae5-a935-7509c28c8867\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") "
Oct 29 22:01:19.922582 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.922463    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-token\" (UniqueName: \"kubernetes.io/projected/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-token\") pod \"3dcff5f2-db63-4ae5-a935-7509c28c8867\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") "
Oct 29 22:01:19.922582 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.922499    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dcff5f2-db63-4ae5-a935-7509c28c8867-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dcff5f2-db63-4ae5-a935-7509c28c8867-istiod-ca-cert\") pod \"3dcff5f2-db63-4ae5-a935-7509c28c8867\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") "
Oct 29 22:01:19.922582 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.922517    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-envoy\") pod \"3dcff5f2-db63-4ae5-a935-7509c28c8867\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") "
Oct 29 22:01:19.922582 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.922544    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-data\") pod \"3dcff5f2-db63-4ae5-a935-7509c28c8867\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") "
Oct 29 22:01:19.922582 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.922568    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-podinfo\") pod \"3dcff5f2-db63-4ae5-a935-7509c28c8867\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") "
Oct 29 22:01:19.922892 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.922668    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-workload-certs" pod "3dcff5f2-db63-4ae5-a935-7509c28c8867" (UID: "3dcff5f2-db63-4ae5-a935-7509c28c8867"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:01:19.922892 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.922693    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-credential-socket\") pod \"3dcff5f2-db63-4ae5-a935-7509c28c8867\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") "
Oct 29 22:01:19.922892 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.922802    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-workload-socket" pod "3dcff5f2-db63-4ae5-a935-7509c28c8867" (UID: "3dcff5f2-db63-4ae5-a935-7509c28c8867"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:01:19.922892 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.922870    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-data" pod "3dcff5f2-db63-4ae5-a935-7509c28c8867" (UID: "3dcff5f2-db63-4ae5-a935-7509c28c8867"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:01:19.923107 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.922905    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dcff5f2-db63-4ae5-a935-7509c28c8867-kube-api-access-hfgd9\" (UniqueName: \"kubernetes.io/projected/3dcff5f2-db63-4ae5-a935-7509c28c8867-kube-api-access-hfgd9\") pod \"3dcff5f2-db63-4ae5-a935-7509c28c8867\" (UID: \"3dcff5f2-db63-4ae5-a935-7509c28c8867\") "
Oct 29 22:01:19.923107 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.922956    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dcff5f2-db63-4ae5-a935-7509c28c8867-istiod-ca-cert" pod "3dcff5f2-db63-4ae5-a935-7509c28c8867" (UID: "3dcff5f2-db63-4ae5-a935-7509c28c8867"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:01:19.923107 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.923096    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-credential-socket" pod "3dcff5f2-db63-4ae5-a935-7509c28c8867" (UID: "3dcff5f2-db63-4ae5-a935-7509c28c8867"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:01:19.923756 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.923729    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:01:19.923756 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.923759    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:01:19.923934 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.923775    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:01:19.923934 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.923790    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dcff5f2-db63-4ae5-a935-7509c28c8867-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:01:19.923934 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.923806    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:01:19.924836 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.924807    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-token" pod "3dcff5f2-db63-4ae5-a935-7509c28c8867" (UID: "3dcff5f2-db63-4ae5-a935-7509c28c8867"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:01:19.924959 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.924853    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-podinfo" pod "3dcff5f2-db63-4ae5-a935-7509c28c8867" (UID: "3dcff5f2-db63-4ae5-a935-7509c28c8867"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:01:19.925061 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.925045    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-envoy" pod "3dcff5f2-db63-4ae5-a935-7509c28c8867" (UID: "3dcff5f2-db63-4ae5-a935-7509c28c8867"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:01:19.925124 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:19.925103    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dcff5f2-db63-4ae5-a935-7509c28c8867-kube-api-access-hfgd9" pod "3dcff5f2-db63-4ae5-a935-7509c28c8867" (UID: "3dcff5f2-db63-4ae5-a935-7509c28c8867"). InnerVolumeSpecName "kube-api-access-hfgd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:01:20.025767 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:20.025668    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:01:20.025767 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:20.025701    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:01:20.025767 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:20.025713    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dcff5f2-db63-4ae5-a935-7509c28c8867-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:01:20.025767 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:20.025725    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfgd9\" (UniqueName: \"kubernetes.io/projected/3dcff5f2-db63-4ae5-a935-7509c28c8867-kube-api-access-hfgd9\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:01:20.080601 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:20.080571    4962 scope.go:122] "RemoveContainer" containerID="dcb6295765a76eb116d2a684331e402fcd7bf11bcfaad16fe2829f4f0dbb605d"
Oct 29 22:01:20.518318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:20.518272    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5" event={"ID":"3dcff5f2-db63-4ae5-a935-7509c28c8867","Type":"ContainerDied","Data":"0525d2fd070b5c66eee7deed566b787b6af935f5633b3a5e62301dae8c921ca5"}
Oct 29 22:01:20.518780 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:20.518349    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk" event={"ID":"029c7ea5-bf31-4b22-96b8-41a693287efe","Type":"ContainerDied","Data":"ccefe3046589b53dd91037b132f41e76cef5be5a2e44d15f9d179d6faf8dbaad"}
Oct 29 22:01:20.518780 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:20.518369    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"
Oct 29 22:01:20.518780 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:20.518379    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"
Oct 29 22:01:20.530532 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:20.530511    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:20.635833 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:20.635807    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"]
Oct 29 22:01:20.645971 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:20.643826    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-zz6dk"]
Oct 29 22:01:20.720518 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:20.720479    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"]
Oct 29 22:01:20.724513 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:20.724481    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tz6f5"]
Oct 29 22:01:21.061824 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:21.061788    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="029c7ea5-bf31-4b22-96b8-41a693287efe" path="/var/lib/kubelet/pods/029c7ea5-bf31-4b22-96b8-41a693287efe/volumes"
Oct 29 22:01:21.063272 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:21.063243    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dcff5f2-db63-4ae5-a935-7509c28c8867" path="/var/lib/kubelet/pods/3dcff5f2-db63-4ae5-a935-7509c28c8867/volumes"
Oct 29 22:01:22.872193 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:01:22.870822    4962 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcff5f2_db63_4ae5_a935_7509c28c8867.slice/crio-conmon-dcb6295765a76eb116d2a684331e402fcd7bf11bcfaad16fe2829f4f0dbb605d.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:01:25.579666 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:25.576403    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:25.625324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:25.620487    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:25.759322 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:25.754890    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:26.069310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:26.068514    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:26.508838 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:26.508782    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:26.776106 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:01:26.776002    4962 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcff5f2_db63_4ae5_a935_7509c28c8867.slice/crio-conmon-dcb6295765a76eb116d2a684331e402fcd7bf11bcfaad16fe2829f4f0dbb605d.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:01:28.067401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:28.067358    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:01:28.069014 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:28.068987    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:28.573312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:28.572648    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:01:28.573312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:28.572754    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:01:28.665661 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:28.665620    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:01:28.724312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:28.688724    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:28.977309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:28.925454    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:01:29.001974 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:29.001626    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:01:29.218473 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:29.201415    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:01:29.218473 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:29.209102    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:01:29.305248 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:29.305147    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:01:29.409310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:29.402092    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": context deadline exceeded"
Oct 29 22:01:29.681368 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:29.608225    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:29.682171 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:29.610022    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:29.759319 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:29.738497    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:29.971540 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:29.971435    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:29.973623 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:29.973584    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:01:30.027964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:30.027909    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:30.248173 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:30.248067    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:01:30.256910 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:30.256866    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:01:30.322578 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:30.320769    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:30.392648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:30.392612    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:01:30.392958 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:30.392925    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57" podUID="67b4a685-bfa5-4a36-adf7-68c7591137c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:30.415767 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:30.415731    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:01:30.416036 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:30.416005    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp" podUID="5d5e71cf-5078-4917-94b8-2b531b143393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:30.446797 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:01:30.446764    4962 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.242s"
Oct 29 22:01:30.475078 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:30.475053    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 22:01:30.496516 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:30.496489    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:01:30.498980 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:30.498056    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:30.499264 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:30.499242    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 22:01:30.523279 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:30.523257    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 22:01:30.524015 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:30.523675    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:01:30.524422 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:30.524405    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 22:01:30.524971 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:30.524955    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:01:30.525160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:30.525137    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 22:01:30.544776 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:30.544730    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:30.683833 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:30.683779    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:01:30.684153 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:30.684101    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:01:30.749598 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:30.749498    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:01:30.871029 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:30.870982    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:31.017369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:31.007112    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:01:31.017369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:31.007513    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:01:31.017369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:31.007784    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": context deadline exceeded"
Oct 29 22:01:31.084313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:31.047849    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:31.255885 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:31.255848    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:31.257255 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:31.257226    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:31.341435 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:31.336376    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:31.685450 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:31.685405    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:31.795312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:31.686268    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:31.869390 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:31.750370    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:31.869390 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:31.822645    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:32.045888 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.045785    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:32.047136 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.047106    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:01:32.094312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.086374    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:32.267452 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.260090    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:01:32.282350 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.272398    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:32.304625 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.304532    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:01:32.304882 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.304852    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z" podUID="132ed775-c9e6-41f1-b224-fff550c6e073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:32.382324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.364423    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:01:32.427314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.398343    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"]
Oct 29 22:01:32.427314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.399437    4962 cpu_manager.go:414] "containerMap: removing container" podUID="3dcff5f2-db63-4ae5-a935-7509c28c8867" containerName="istio-proxy"
Oct 29 22:01:32.427314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.399460    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dcff5f2-db63-4ae5-a935-7509c28c8867" containerName="istio-proxy"
Oct 29 22:01:32.427314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.399483    4962 cpu_manager.go:414] "containerMap: removing container" podUID="029c7ea5-bf31-4b22-96b8-41a693287efe" containerName="istio-proxy"
Oct 29 22:01:32.427314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.399492    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="029c7ea5-bf31-4b22-96b8-41a693287efe" containerName="istio-proxy"
Oct 29 22:01:32.427314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.399709    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="029c7ea5-bf31-4b22-96b8-41a693287efe" containerName="istio-proxy"
Oct 29 22:01:32.427314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.399741    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dcff5f2-db63-4ae5-a935-7509c28c8867" containerName="istio-proxy"
Oct 29 22:01:32.432665 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.432635    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:01:32.476620 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.476584    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e4e24e\"/\"openshift-service-ca.crt\""
Oct 29 22:01:32.477011 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.476989    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e4e24e\"/\"default-dockercfg-8bczv\""
Oct 29 22:01:32.477263 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.477246    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-e4e24e\"/\"distinct-2-w0-e4e24e-kserve-self-signed-certs\""
Oct 29 22:01:32.477487 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.477471    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-e4e24e\"/\"kube-root-ca.crt\""
Oct 29 22:01:32.560408 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.533248    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksd44\" (UniqueName: \"kubernetes.io/projected/1e7bb25d-994d-4600-8ade-890853b6f0d3-kube-api-access-ksd44\") pod \"distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j\" (UID: \"1e7bb25d-994d-4600-8ade-890853b6f0d3\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:01:32.673403 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.659180    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-tmp-dir\") pod \"distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j\" (UID: \"1e7bb25d-994d-4600-8ade-890853b6f0d3\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:01:32.673403 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.659351    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-dshm\") pod \"distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j\" (UID: \"1e7bb25d-994d-4600-8ade-890853b6f0d3\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:01:32.673403 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.659392    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e7bb25d-994d-4600-8ade-890853b6f0d3-tls-certs\") pod \"distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j\" (UID: \"1e7bb25d-994d-4600-8ade-890853b6f0d3\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:01:32.673403 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.659496    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-home\") pod \"distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j\" (UID: \"1e7bb25d-994d-4600-8ade-890853b6f0d3\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:01:32.673403 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.659554    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-model-cache\") pod \"distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j\" (UID: \"1e7bb25d-994d-4600-8ade-890853b6f0d3\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:01:32.673403 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.588419    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:32.693494 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.693432    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:01:32.739121 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.739074    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:01:32.739466 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.739401    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv" podUID="d71de0ec-1eaa-481f-8cee-cb256e330136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:32.739683 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.739650    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"]
Oct 29 22:01:32.762631 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.762593    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-tmp-dir\") pod \"distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j\" (UID: \"1e7bb25d-994d-4600-8ade-890853b6f0d3\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:01:32.768513 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.768478    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-dshm\") pod \"distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j\" (UID: \"1e7bb25d-994d-4600-8ade-890853b6f0d3\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:01:32.768751 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.768732    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e7bb25d-994d-4600-8ade-890853b6f0d3-tls-certs\") pod \"distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j\" (UID: \"1e7bb25d-994d-4600-8ade-890853b6f0d3\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:01:32.776313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.768943    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-home\") pod \"distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j\" (UID: \"1e7bb25d-994d-4600-8ade-890853b6f0d3\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:01:32.776643 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.776616    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-model-cache\") pod \"distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j\" (UID: \"1e7bb25d-994d-4600-8ade-890853b6f0d3\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:01:32.787950 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.787921    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-model-cache\") pod \"distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j\" (UID: \"1e7bb25d-994d-4600-8ade-890853b6f0d3\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:01:32.820021 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.819844    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksd44\" (UniqueName: \"kubernetes.io/projected/1e7bb25d-994d-4600-8ade-890853b6f0d3-kube-api-access-ksd44\") pod \"distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j\" (UID: \"1e7bb25d-994d-4600-8ade-890853b6f0d3\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:01:32.831424 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.831378    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-home\") pod \"distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j\" (UID: \"1e7bb25d-994d-4600-8ade-890853b6f0d3\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:01:32.847548 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.820799    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-tmp-dir\") pod \"distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j\" (UID: \"1e7bb25d-994d-4600-8ade-890853b6f0d3\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:01:32.848042 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.848010    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:32.928315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.902412    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e7bb25d-994d-4600-8ade-890853b6f0d3-tls-certs\") pod \"distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j\" (UID: \"1e7bb25d-994d-4600-8ade-890853b6f0d3\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:01:32.987877 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.929577    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:01:32.988442 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.932382    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksd44\" (UniqueName: \"kubernetes.io/projected/1e7bb25d-994d-4600-8ade-890853b6f0d3-kube-api-access-ksd44\") pod \"distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j\" (UID: \"1e7bb25d-994d-4600-8ade-890853b6f0d3\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:01:32.988686 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:32.932589    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-dshm\") pod \"distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j\" (UID: \"1e7bb25d-994d-4600-8ade-890853b6f0d3\") " pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:01:33.052218 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:33.052165    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:01:33.109015 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:33.055900    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:33.161479 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:33.134867    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:33.299313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:33.286034    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:01:33.299313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:33.286900    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:01:33.299313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:33.288698    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:01:33.377491 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:33.377368    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:33.820312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:33.807787    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:33.869327 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:33.852541    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:33.991521 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:33.991480    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:01:34.099888 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:34.099840    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.108835 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:34.108793    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.240029 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:34.239984    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.254432 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:34.254270    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:01:34.254747 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:34.254707    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt" podUID="730edf47-8b07-477a-acfc-47faaaacfc84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.275416 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:34.275367    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.288623 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:34.288580    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.289175 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:34.289146    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:01:34.431312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:34.429622    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.431312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:34.430404    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:01:34.431312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:34.430450    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz" podUID="8054bc19-6db7-462f-8af0-335903ce62f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.681311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:34.677822    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:01:34.681311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:34.678184    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42" podUID="e1897161-0016-400a-9928-8b407f4148e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.809434 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:34.809370    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:34.856413 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:34.856368    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:35.051389 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:35.029584    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:35.106864 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:35.106824    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:35.163135 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:35.163080    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:01:35.242340 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:35.241680    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:01:35.342135 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:35.342086    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:01:35.349467 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:35.349428    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:01:35.373694 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:35.358273    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:01:35.373694 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:35.358354    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf" podUID="75cf3e4f-0488-4210-ab72-17b7afdeda6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:35.435930 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:35.435886    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:35.762747 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:35.762699    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:01:35.763068 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:35.763019    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5" podUID="a437f4e2-dc10-41de-851a-ee488b08c6f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:35.780482 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:35.780432    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:35.887325 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:35.882313    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:01:36.095378 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:36.080063    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:36.137339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:36.137277    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:01:36.137869 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:36.137823    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:01:36.192065 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:36.190577    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:36.256314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:36.237568    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"]
Oct 29 22:01:36.259471 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:36.259427    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:01:36.431777 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:36.431735    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:01:36.452410 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:36.439469    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:01:36.494852 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:36.453723    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:36.868096 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:36.868067    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:01:36.892222 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:36.882479    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:37.164382 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:37.119644    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:37.164382 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:37.145435    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:37.164382 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:37.145775    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:01:37.278326 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:37.275090    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 22:01:37.278326 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:37.275149    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:01:37.327546 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:37.327205    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 22:01:37.460377 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:37.455264    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:01:37.496932 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:37.496869    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:01:37.859147 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:37.859114    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 22:01:37.956873 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:37.956830    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:01:38.017107 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:38.017075    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 22:01:38.019207 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:38.019176    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" event={"ID":"1e7bb25d-994d-4600-8ade-890853b6f0d3","Type":"ContainerStarted","Data":"cbc22d21fdd53d80e227fb3635cf5f94eda9aea7dabce1eadc310c4e0ea46c8a"}
Oct 29 22:01:38.078545 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:38.078515    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:01:38.100748 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:38.100719    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:01:38.104528 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:38.104502    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 22:01:38.120231 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:01:38.120137    4962 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcff5f2_db63_4ae5_a935_7509c28c8867.slice/crio-conmon-dcb6295765a76eb116d2a684331e402fcd7bf11bcfaad16fe2829f4f0dbb605d.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:01:38.120725 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:01:38.120688    4962 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcff5f2_db63_4ae5_a935_7509c28c8867.slice/crio-conmon-dcb6295765a76eb116d2a684331e402fcd7bf11bcfaad16fe2829f4f0dbb605d.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:01:38.158048 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:38.157986    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podStartSLOduration=6.157967791 podStartE2EDuration="6.157967791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:01:38.153766244 +0000 UTC m=+919.226723211" watchObservedRunningTime="2026-10-29 22:01:38.157967791 +0000 UTC m=+919.230924757"
Oct 29 22:01:38.306907 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:38.306879    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:01:39.027245 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:39.027208    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" event={"ID":"1e7bb25d-994d-4600-8ade-890853b6f0d3","Type":"ContainerStarted","Data":"7bdb41fc2ff92c2ea0ac3dc8e850a794cd4a87550bc77654fac522041e223711"}
Oct 29 22:01:39.733384 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:01:39.733344    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:01:43.307218 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:43.307187    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:01:43.319346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:43.319321    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:01:44.074347 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:01:44.074315    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:01:48.219398 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:01:48.219363    4962 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcff5f2_db63_4ae5_a935_7509c28c8867.slice/crio-conmon-dcb6295765a76eb116d2a684331e402fcd7bf11bcfaad16fe2829f4f0dbb605d.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:01:52.704697 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:01:52.704658    4962 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcff5f2_db63_4ae5_a935_7509c28c8867.slice/crio-conmon-dcb6295765a76eb116d2a684331e402fcd7bf11bcfaad16fe2829f4f0dbb605d.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:01:58.305264 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:01:58.305225    4962 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcff5f2_db63_4ae5_a935_7509c28c8867.slice/crio-conmon-dcb6295765a76eb116d2a684331e402fcd7bf11bcfaad16fe2829f4f0dbb605d.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:02:01.476062 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:01.476025    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz"]
Oct 29 22:02:01.481002 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:01.480976    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz"
Oct 29 22:02:01.494728 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:01.494703    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd29dac-756d-4875-8fa3-3f9662b139e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz\" (UID: \"8cd29dac-756d-4875-8fa3-3f9662b139e2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz"
Oct 29 22:02:01.494846 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:01.494747    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n9qn\" (UniqueName: \"kubernetes.io/projected/8cd29dac-756d-4875-8fa3-3f9662b139e2-kube-api-access-6n9qn\") pod \"payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz\" (UID: \"8cd29dac-756d-4875-8fa3-3f9662b139e2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz"
Oct 29 22:02:01.502951 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:01.502925    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz"]
Oct 29 22:02:01.528837 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:01.528790    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-751a7416\""
Oct 29 22:02:01.537446 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:01.533361    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-751a7416-dockercfg-qjfb8\""
Oct 29 22:02:01.780049 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:01.779939    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd29dac-756d-4875-8fa3-3f9662b139e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz\" (UID: \"8cd29dac-756d-4875-8fa3-3f9662b139e2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz"
Oct 29 22:02:01.780347 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:01.780327    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n9qn\" (UniqueName: \"kubernetes.io/projected/8cd29dac-756d-4875-8fa3-3f9662b139e2-kube-api-access-6n9qn\") pod \"payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz\" (UID: \"8cd29dac-756d-4875-8fa3-3f9662b139e2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz"
Oct 29 22:02:01.802505 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:01.800054    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd29dac-756d-4875-8fa3-3f9662b139e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz\" (UID: \"8cd29dac-756d-4875-8fa3-3f9662b139e2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz"
Oct 29 22:02:01.894311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:01.892361    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n9qn\" (UniqueName: \"kubernetes.io/projected/8cd29dac-756d-4875-8fa3-3f9662b139e2-kube-api-access-6n9qn\") pod \"payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz\" (UID: \"8cd29dac-756d-4875-8fa3-3f9662b139e2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz"
Oct 29 22:02:02.095086 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:02.094975    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz"
Oct 29 22:02:02.409936 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:02.409905    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz"]
Oct 29 22:02:02.410232 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:02:02.410210    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd29dac_756d_4875_8fa3_3f9662b139e2.slice/crio-4c7209e54be72a2c94a3d74aec058266a4ecf727b6fa203d00fcc142d54a8d4f WatchSource:0}: Error finding container 4c7209e54be72a2c94a3d74aec058266a4ecf727b6fa203d00fcc142d54a8d4f: Status 404 returned error can't find the container with id 4c7209e54be72a2c94a3d74aec058266a4ecf727b6fa203d00fcc142d54a8d4f
Oct 29 22:02:03.279309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:03.279253    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz" event={"ID":"8cd29dac-756d-4875-8fa3-3f9662b139e2","Type":"ContainerStarted","Data":"07fce3c11363df39133e430a34603873e6b06a706d635f34a541e1bd23d2c1f9"}
Oct 29 22:02:03.281232 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:03.281191    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz" event={"ID":"8cd29dac-756d-4875-8fa3-3f9662b139e2","Type":"ContainerStarted","Data":"4c7209e54be72a2c94a3d74aec058266a4ecf727b6fa203d00fcc142d54a8d4f"}
Oct 29 22:02:03.281487 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:03.281469    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz"
Oct 29 22:02:03.342759 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:03.342661    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz" podStartSLOduration=2.342635224 podStartE2EDuration="2.342635224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:03.334613874 +0000 UTC m=+944.407570863" watchObservedRunningTime="2026-10-29 22:02:03.342635224 +0000 UTC m=+944.415592189"
Oct 29 22:02:06.475047 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:02:06.475004    4962 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcff5f2_db63_4ae5_a935_7509c28c8867.slice/crio-conmon-dcb6295765a76eb116d2a684331e402fcd7bf11bcfaad16fe2829f4f0dbb605d.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:02:06.475641 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:02:06.475104    4962 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcff5f2_db63_4ae5_a935_7509c28c8867.slice/crio-conmon-dcb6295765a76eb116d2a684331e402fcd7bf11bcfaad16fe2829f4f0dbb605d.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:02:07.696362 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:02:07.696329    4962 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcff5f2_db63_4ae5_a935_7509c28c8867.slice/crio-conmon-dcb6295765a76eb116d2a684331e402fcd7bf11bcfaad16fe2829f4f0dbb605d.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:02:08.321768 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:02:08.321728    4962 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcff5f2_db63_4ae5_a935_7509c28c8867.slice/crio-conmon-dcb6295765a76eb116d2a684331e402fcd7bf11bcfaad16fe2829f4f0dbb605d.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:02:09.721705 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:02:09.721663    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:02:12.974930 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:12.974895    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k"]
Oct 29 22:02:12.979570 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:12.979546    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k"
Oct 29 22:02:12.996653 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:12.996627    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-751a7416-serving-cert\""
Oct 29 22:02:13.017203 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:13.017177    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k"]
Oct 29 22:02:13.083850 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:13.083811    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c7978e3-558b-4148-b0fc-3e275473a211-metrics-certs\") pod \"maas-api-e2e-mt-751a7416-56467cbd84-k474k\" (UID: \"3c7978e3-558b-4148-b0fc-3e275473a211\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k"
Oct 29 22:02:13.084000 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:13.083859    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c7978e3-558b-4148-b0fc-3e275473a211-maas-api-tls\") pod \"maas-api-e2e-mt-751a7416-56467cbd84-k474k\" (UID: \"3c7978e3-558b-4148-b0fc-3e275473a211\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k"
Oct 29 22:02:13.084065 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:13.084048    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krmvt\" (UniqueName: \"kubernetes.io/projected/3c7978e3-558b-4148-b0fc-3e275473a211-kube-api-access-krmvt\") pod \"maas-api-e2e-mt-751a7416-56467cbd84-k474k\" (UID: \"3c7978e3-558b-4148-b0fc-3e275473a211\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k"
Oct 29 22:02:13.084161 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:13.084148    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7978e3-558b-4148-b0fc-3e275473a211-openshift-service-ca\") pod \"maas-api-e2e-mt-751a7416-56467cbd84-k474k\" (UID: \"3c7978e3-558b-4148-b0fc-3e275473a211\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k"
Oct 29 22:02:13.204034 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:13.197926    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krmvt\" (UniqueName: \"kubernetes.io/projected/3c7978e3-558b-4148-b0fc-3e275473a211-kube-api-access-krmvt\") pod \"maas-api-e2e-mt-751a7416-56467cbd84-k474k\" (UID: \"3c7978e3-558b-4148-b0fc-3e275473a211\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k"
Oct 29 22:02:13.204500 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:13.204471    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7978e3-558b-4148-b0fc-3e275473a211-openshift-service-ca\") pod \"maas-api-e2e-mt-751a7416-56467cbd84-k474k\" (UID: \"3c7978e3-558b-4148-b0fc-3e275473a211\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k"
Oct 29 22:02:13.231447 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:13.210930    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7978e3-558b-4148-b0fc-3e275473a211-openshift-service-ca\") pod \"maas-api-e2e-mt-751a7416-56467cbd84-k474k\" (UID: \"3c7978e3-558b-4148-b0fc-3e275473a211\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k"
Oct 29 22:02:13.231940 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:13.231912    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c7978e3-558b-4148-b0fc-3e275473a211-metrics-certs\") pod \"maas-api-e2e-mt-751a7416-56467cbd84-k474k\" (UID: \"3c7978e3-558b-4148-b0fc-3e275473a211\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k"
Oct 29 22:02:13.232172 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:13.232132    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c7978e3-558b-4148-b0fc-3e275473a211-maas-api-tls\") pod \"maas-api-e2e-mt-751a7416-56467cbd84-k474k\" (UID: \"3c7978e3-558b-4148-b0fc-3e275473a211\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k"
Oct 29 22:02:13.241656 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:13.241592    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c7978e3-558b-4148-b0fc-3e275473a211-maas-api-tls\") pod \"maas-api-e2e-mt-751a7416-56467cbd84-k474k\" (UID: \"3c7978e3-558b-4148-b0fc-3e275473a211\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k"
Oct 29 22:02:13.250451 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:13.250408    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c7978e3-558b-4148-b0fc-3e275473a211-metrics-certs\") pod \"maas-api-e2e-mt-751a7416-56467cbd84-k474k\" (UID: \"3c7978e3-558b-4148-b0fc-3e275473a211\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k"
Oct 29 22:02:13.250962 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:13.250932    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krmvt\" (UniqueName: \"kubernetes.io/projected/3c7978e3-558b-4148-b0fc-3e275473a211-kube-api-access-krmvt\") pod \"maas-api-e2e-mt-751a7416-56467cbd84-k474k\" (UID: \"3c7978e3-558b-4148-b0fc-3e275473a211\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k"
Oct 29 22:02:13.302315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:13.296588    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k"
Oct 29 22:02:13.678800 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:02:13.678763    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c7978e3_558b_4148_b0fc_3e275473a211.slice/crio-ef5a34c6f34edc0c308dc83bee2d549acec9515792f74374d486c1c5532c9e4d WatchSource:0}: Error finding container ef5a34c6f34edc0c308dc83bee2d549acec9515792f74374d486c1c5532c9e4d: Status 404 returned error can't find the container with id ef5a34c6f34edc0c308dc83bee2d549acec9515792f74374d486c1c5532c9e4d
Oct 29 22:02:13.679651 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:13.679627    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k"]
Oct 29 22:02:14.625641 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:14.623486    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k" event={"ID":"3c7978e3-558b-4148-b0fc-3e275473a211","Type":"ContainerStarted","Data":"91693487f5669a230f61e5d972d8dff350fdb629c9d55173532e7cefc11e2e14"}
Oct 29 22:02:14.625641 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:14.623540    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k" event={"ID":"3c7978e3-558b-4148-b0fc-3e275473a211","Type":"ContainerStarted","Data":"ef5a34c6f34edc0c308dc83bee2d549acec9515792f74374d486c1c5532c9e4d"}
Oct 29 22:02:14.625641 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:14.624747    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k"
Oct 29 22:02:14.692040 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:14.691988    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k" podStartSLOduration=2.204962723 podStartE2EDuration="2.69197361s" totalImagesPullingTime="487.010887ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:14.690140938 +0000 UTC m=+955.763097919" watchObservedRunningTime="2026-10-29 22:02:14.69197361 +0000 UTC m=+955.764930573"
Oct 29 22:02:15.295668 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:15.295637    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz"
Oct 29 22:02:18.344669 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:02:18.344633    4962 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcff5f2_db63_4ae5_a935_7509c28c8867.slice/crio-conmon-dcb6295765a76eb116d2a684331e402fcd7bf11bcfaad16fe2829f4f0dbb605d.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:02:21.642365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:21.642331    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k"
Oct 29 22:02:28.712484 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:28.712446    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7"]
Oct 29 22:02:28.717354 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:28.717338    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7"
Oct 29 22:02:28.741022 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:28.740994    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7"]
Oct 29 22:02:28.747833 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:28.747810    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a9f9aa\""
Oct 29 22:02:28.747957 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:28.747897    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a9f9aa-dockercfg-wxxbw\""
Oct 29 22:02:28.771142 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:28.771119    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt"]
Oct 29 22:02:28.775600 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:28.775582    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt"
Oct 29 22:02:28.796342 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:28.796317    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt"]
Oct 29 22:02:29.135622 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:29.135537    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6gbs\" (UniqueName: \"kubernetes.io/projected/f74f6919-d222-4c09-8b7d-6a1aa56229a3-kube-api-access-c6gbs\") pod \"payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt\" (UID: \"f74f6919-d222-4c09-8b7d-6a1aa56229a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt"
Oct 29 22:02:29.135892 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:29.135871    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km92m\" (UniqueName: \"kubernetes.io/projected/c01b8e41-5665-45da-85ff-8ca4b229b30e-kube-api-access-km92m\") pod \"payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7\" (UID: \"c01b8e41-5665-45da-85ff-8ca4b229b30e\") " pod="openshift-ingress/payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7"
Oct 29 22:02:29.136260 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:29.136237    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f74f6919-d222-4c09-8b7d-6a1aa56229a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt\" (UID: \"f74f6919-d222-4c09-8b7d-6a1aa56229a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt"
Oct 29 22:02:29.136696 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:29.136675    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c01b8e41-5665-45da-85ff-8ca4b229b30e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7\" (UID: \"c01b8e41-5665-45da-85ff-8ca4b229b30e\") " pod="openshift-ingress/payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7"
Oct 29 22:02:29.241143 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:29.241101    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c01b8e41-5665-45da-85ff-8ca4b229b30e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7\" (UID: \"c01b8e41-5665-45da-85ff-8ca4b229b30e\") " pod="openshift-ingress/payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7"
Oct 29 22:02:29.241364 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:29.241344    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6gbs\" (UniqueName: \"kubernetes.io/projected/f74f6919-d222-4c09-8b7d-6a1aa56229a3-kube-api-access-c6gbs\") pod \"payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt\" (UID: \"f74f6919-d222-4c09-8b7d-6a1aa56229a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt"
Oct 29 22:02:29.241468 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:29.241400    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km92m\" (UniqueName: \"kubernetes.io/projected/c01b8e41-5665-45da-85ff-8ca4b229b30e-kube-api-access-km92m\") pod \"payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7\" (UID: \"c01b8e41-5665-45da-85ff-8ca4b229b30e\") " pod="openshift-ingress/payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7"
Oct 29 22:02:29.241541 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:29.241520    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f74f6919-d222-4c09-8b7d-6a1aa56229a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt\" (UID: \"f74f6919-d222-4c09-8b7d-6a1aa56229a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt"
Oct 29 22:02:29.241766 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:29.241741    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c01b8e41-5665-45da-85ff-8ca4b229b30e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7\" (UID: \"c01b8e41-5665-45da-85ff-8ca4b229b30e\") " pod="openshift-ingress/payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7"
Oct 29 22:02:29.242213 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:29.242189    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f74f6919-d222-4c09-8b7d-6a1aa56229a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt\" (UID: \"f74f6919-d222-4c09-8b7d-6a1aa56229a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt"
Oct 29 22:02:29.265246 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:29.265224    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km92m\" (UniqueName: \"kubernetes.io/projected/c01b8e41-5665-45da-85ff-8ca4b229b30e-kube-api-access-km92m\") pod \"payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7\" (UID: \"c01b8e41-5665-45da-85ff-8ca4b229b30e\") " pod="openshift-ingress/payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7"
Oct 29 22:02:29.265356 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:29.265312    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6gbs\" (UniqueName: \"kubernetes.io/projected/f74f6919-d222-4c09-8b7d-6a1aa56229a3-kube-api-access-c6gbs\") pod \"payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt\" (UID: \"f74f6919-d222-4c09-8b7d-6a1aa56229a3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt"
Oct 29 22:02:29.332341 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:29.332316    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7"
Oct 29 22:02:29.399617 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:29.399582    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt"
Oct 29 22:02:29.723674 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:29.723645    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7"]
Oct 29 22:02:29.738437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:29.738412    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt"]
Oct 29 22:02:29.739685 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:02:29.739660    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf74f6919_d222_4c09_8b7d_6a1aa56229a3.slice/crio-1b37ea6733f6dba285034dcadc63119c4504da812528c49d01408977ce54ca42 WatchSource:0}: Error finding container 1b37ea6733f6dba285034dcadc63119c4504da812528c49d01408977ce54ca42: Status 404 returned error can't find the container with id 1b37ea6733f6dba285034dcadc63119c4504da812528c49d01408977ce54ca42
Oct 29 22:02:30.215149 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:30.215113    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7" event={"ID":"c01b8e41-5665-45da-85ff-8ca4b229b30e","Type":"ContainerStarted","Data":"2f0f07096a30bd3c685f4a13454618caf51b3eb22bf3e4398eadd67a55acb233"}
Oct 29 22:02:30.215368 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:30.215165    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7" event={"ID":"c01b8e41-5665-45da-85ff-8ca4b229b30e","Type":"ContainerStarted","Data":"eb0aa4919974588b579dc5676feaed1f415275a389170649cfa8f2f0c963f5c1"}
Oct 29 22:02:30.215368 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:30.215194    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7"
Oct 29 22:02:30.216754 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:30.216732    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt" event={"ID":"f74f6919-d222-4c09-8b7d-6a1aa56229a3","Type":"ContainerStarted","Data":"3abcdff572f061b07a70368e1713b96bc4019bcb0c59a79bb192b7dff0de83ac"}
Oct 29 22:02:30.216875 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:30.216769    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt" event={"ID":"f74f6919-d222-4c09-8b7d-6a1aa56229a3","Type":"ContainerStarted","Data":"1b37ea6733f6dba285034dcadc63119c4504da812528c49d01408977ce54ca42"}
Oct 29 22:02:30.216875 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:30.216854    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt"
Oct 29 22:02:30.261146 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:30.261089    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7" podStartSLOduration=2.261076154 podStartE2EDuration="2.261076154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:30.257162389 +0000 UTC m=+971.330119353" watchObservedRunningTime="2026-10-29 22:02:30.261076154 +0000 UTC m=+971.334033118"
Oct 29 22:02:30.299917 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:30.299867    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt" podStartSLOduration=2.299854693 podStartE2EDuration="2.299854693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:02:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:02:30.298113836 +0000 UTC m=+971.371070808" watchObservedRunningTime="2026-10-29 22:02:30.299854693 +0000 UTC m=+971.372811693"
Oct 29 22:02:39.721256 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:02:39.721220    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:02:41.283766 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:41.283736    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7"
Oct 29 22:02:41.284187 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:41.283801    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt"
Oct 29 22:02:53.096313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:53.074390    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:02:53.106311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:53.099679    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:53.355992 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:53.353003    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:53.380542 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:53.380478    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:53.383923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:53.383872    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:02:53.406671 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:53.406639    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:53.701467 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:53.701413    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:02:53.794181 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:53.794138    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:53.919502 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:53.919438    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:54.081404 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:54.071784    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:54.109385 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:54.101374    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:54.133592 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:54.101414    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:02:54.134389 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:54.134228    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:02:54.134389 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:54.134318    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:02:54.359366 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:54.358360    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:54.427105 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:54.398113    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 22:02:54.427105 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:54.399110    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:54.427105 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:54.413358    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:54.447316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:54.443438    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 22:02:54.447316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:54.443511    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 22:02:54.470500 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:54.470436    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:54.708314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:54.705559    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:54.708314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:54.706226    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:02:54.799342 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:54.798728    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:02:54.799342 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:54.799244    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 22:02:54.909710 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:54.909671    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:02:54.913479 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:54.913441    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh" podUID="f06b758b-d09f-4882-a198-d88e91c7f390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:54.929953 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:54.929910    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:54.930446 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:54.930420    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 22:02:55.144153 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:55.144058    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:55.234308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:55.165181    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:02:55.234308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:55.166656    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:02:55.360312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:55.359437    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:55.414328 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:55.414012    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:55.462816 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:55.414125    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:55.463475 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:55.414179    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:55.488629 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:55.414391    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:02:55.488924 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:55.488875    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr" podUID="4ef87fe2-429a-4393-8ddf-fa43e2dd9044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:55.799846 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:55.755515    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded"
Oct 29 22:02:55.799846 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:55.758672    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:55.819443 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:55.819392    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:55.895272 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:55.895239    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:02:55.945456 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:55.945416    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:56.186558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:56.166074    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:56.221354 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:56.216503    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:56.221354 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:56.218378    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:56.393594 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:56.393538    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:02:56.466491 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:56.466389    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:56.498464 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:56.498399    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:56.542410 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:56.542356    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:02:56.542714 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:56.542668    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5" podUID="9762d8d0-a759-42ac-bc1b-c948adfa6f9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:56.649311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:56.646448    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:56.649311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:56.647193    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:02:56.767368 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:56.760512    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:02:56.792473 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:56.792435    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:02:56.792776 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:56.792742    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds" podUID="2a71a069-94f9-4736-8b9f-d43292554254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:56.851342 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:56.851279    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:02:56.956415 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:56.956368    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:57.205264 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:57.205222    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:02:57.262313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:57.219379    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:57.262313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:57.219745    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:02:57.262313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:57.219786    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw" podUID="f6631eff-5d71-483c-aac7-8d578274ce76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:57.262313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:57.220474    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:57.262313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:57.249761    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:02:57.262313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:57.249821    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg" podUID="16540bea-446b-46d9-9cde-4c9c34785ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:57.473388 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:57.470244    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:57.494314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:57.490881    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:57.548324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:57.544940    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:02:57.664309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:57.652527    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:02:57.764646 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:57.764524    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:02:57.860579 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:57.860529    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:02:57.885312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:57.874405    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.004313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:58.003365    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.213422 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:58.213372    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.220393 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:58.220356    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:02:58.236416 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:58.236370    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:02:58.472373 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:58.471398    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.494433 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:58.494387    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.549310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:58.548703    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.744387 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:58.743384    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.776312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:58.771385    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:58.866373 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:58.865386    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:59.011394 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:59.010422    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:59.105065 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:59.105015    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:59.221488 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:59.221439    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:59.223811 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:59.222072    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:59.238411 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:59.238369    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:59.472432 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:59.472383    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:59.501018 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:59.500954    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:02:59.550358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:59.549388    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:02:59.611883 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:59.611518    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:02:59.611883 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:59.611585    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr" podUID="10c5ec48-f48a-4703-a82c-9a6995506125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:59.752394 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:59.745624    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:02:59.773321 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:59.772442    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:02:59.870510 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:02:59.870445    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:03:00.010108 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:00.010001    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:03:00.010441 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:00.010397    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6" podUID="64a76b6c-1542-4fb1-9730-7b770c452b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.011084 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:00.010626    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.224018 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:00.223977    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.229309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:00.225444    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.241511 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:00.241466    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.373540 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:00.373418    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:03:00.373824 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:00.373777    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp" podUID="5d5e71cf-5078-4917-94b8-2b531b143393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.414491 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:00.414437    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.476478 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:00.476433    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:03:00.503435 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:00.503386    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.559374 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:00.558384    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.641879 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:00.641385    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.750005 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:00.749959    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.772653 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:00.772608    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.807433 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:00.807385    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:00.877446 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:00.877394    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.015429 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:01.015385    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.224690 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:01.224641    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.230426 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:01.230368    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.245907 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:01.245858    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.478470 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:01.478421    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.535331 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:01.512589    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.573345 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:01.558957    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.818397 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:01.794895    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.818397 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:01.808091    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:01.843337 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:01.835857    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:03:01.910966 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:01.910920    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:02.032122 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:02.032073    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:02.137944 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:02.137826    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:03:02.138212 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:02.138176    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6" podUID="295e3d96-ff42-4be3-b4de-cd85ca4d8578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:02.283317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:02.244353    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:02.283317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:02.245678    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:03:02.283317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:02.245740    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j" podUID="590ad9cd-3237-4783-8dea-82e67274b580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:02.283317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:02.245816    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:03:02.283317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:02.269163    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:02.337310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:02.326925    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:03:02.337310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:02.327099    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z" podUID="132ed775-c9e6-41f1-b224-fff550c6e073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:02.586314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:02.582147    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:03:02.758749 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:02.692064    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:02.759343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:02.692103    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:02.759760 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:02.737594    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:03:02.841371 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:02.840626    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:03:02.853310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:02.842161    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv" podUID="d71de0ec-1eaa-481f-8cee-cb256e330136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:02.916355 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:02.863702    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:02.923320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:02.877047    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:03:02.923320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:02.916652    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:03.124471 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:03.124372    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:03:03.254408 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:03.254355    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:03.310543 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:03.263380    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:03:03.310543 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:03.280356    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:03:03.352419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:03.352380    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:03:03.352702 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:03.352670    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2" podUID="03b2ca39-a1c7-45bf-8458-f23215d334b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:03.762422 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:03.762373    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:03.763405 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:03.763370    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:03.764188 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:03.764158    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:03.937316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:03.921215    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:03.937316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:03.921698    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:03.937316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:03.921792    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:03:04.170726 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:04.170676    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:04.279898 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:04.279848    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:04.330333 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:04.300761    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:03:04.339320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:04.337378    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:03:04.455796 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:04.339665    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:03:04.456304 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:04.339736    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:04.770863 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:04.769418    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:04.770863 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:04.769820    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:03:04.770863 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:04.770116    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:03:04.926889 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:04.926846    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:03:04.931648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:04.927200    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:04.952322 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:04.947487    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:03:05.177319 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:05.171916    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:03:05.226432 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:05.225628    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 22:03:05.335350 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:05.335271    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:05.407501 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:05.407455    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:03:05.408014 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:05.407863    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 22:03:05.419327 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:05.416683    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 22:03:05.449696 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:05.449637    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:03:05.459273 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:05.457709    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:03:05.698211 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:05.698184    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:03:05.703075 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:05.703001    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 22:03:05.724302 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:05.724266    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:03:05.724523 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:05.724510    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 22:03:05.730598 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:05.729995    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:03:09.722557 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:03:09.722520    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:03:11.190411 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.190373    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"]
Oct 29 22:03:11.194507 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.194482    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.209932 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.209898    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-751a7416-openshift-default-dockercfg-sx494\""
Oct 29 22:03:11.227305 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.226504    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"]
Oct 29 22:03:11.296715 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.296688    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-istio-data\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.296872 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.296821    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-istio-envoy\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.296872 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.296848    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728837f8-b92c-4025-b332-663f29e1570b-istiod-ca-cert\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.296963 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.296917    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728837f8-b92c-4025-b332-663f29e1570b-istio-podinfo\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.297120 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.297103    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-credential-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.297206 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.297141    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728837f8-b92c-4025-b332-663f29e1570b-istio-token\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.297249 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.297215    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwnnc\" (UniqueName: \"kubernetes.io/projected/728837f8-b92c-4025-b332-663f29e1570b-kube-api-access-xwnnc\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.297319 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.297304    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-workload-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.297424 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.297409    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-workload-certs\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.398949 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.398925    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-istio-envoy\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.398949 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.398952    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728837f8-b92c-4025-b332-663f29e1570b-istiod-ca-cert\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.399145 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.398973    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728837f8-b92c-4025-b332-663f29e1570b-istio-podinfo\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.399145 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.399094    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-credential-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.399145 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.399115    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728837f8-b92c-4025-b332-663f29e1570b-istio-token\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.399330 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.399146    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwnnc\" (UniqueName: \"kubernetes.io/projected/728837f8-b92c-4025-b332-663f29e1570b-kube-api-access-xwnnc\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.399330 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.399238    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-workload-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.399529 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.399505    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-workload-certs\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.399632 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.399523    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-credential-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.399632 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.399584    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-istio-data\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.399632 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.399619    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-workload-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.399797 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.399770    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-workload-certs\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.399855 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.399839    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728837f8-b92c-4025-b332-663f29e1570b-istiod-ca-cert\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.399947 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.399929    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-istio-data\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.401421 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.401401    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-istio-envoy\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.401625 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.401608    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728837f8-b92c-4025-b332-663f29e1570b-istio-podinfo\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.421921 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.421892    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwnnc\" (UniqueName: \"kubernetes.io/projected/728837f8-b92c-4025-b332-663f29e1570b-kube-api-access-xwnnc\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.422008 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.421990    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728837f8-b92c-4025-b332-663f29e1570b-istio-token\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-dkstw\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.510944 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.510890    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:11.842896 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.842867    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"]
Oct 29 22:03:11.843276 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:03:11.843252    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod728837f8_b92c_4025_b332_663f29e1570b.slice/crio-2480f00b1eaa05d888287efc332939fa50f1a71439b07809eef0ddb76e1e5499 WatchSource:0}: Error finding container 2480f00b1eaa05d888287efc332939fa50f1a71439b07809eef0ddb76e1e5499: Status 404 returned error can't find the container with id 2480f00b1eaa05d888287efc332939fa50f1a71439b07809eef0ddb76e1e5499
Oct 29 22:03:11.845811 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.845784    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:03:11.845877 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.845845    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:03:11.845877 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:11.845872    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:03:12.059986 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:12.059899    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw" event={"ID":"728837f8-b92c-4025-b332-663f29e1570b","Type":"ContainerStarted","Data":"31e43a8fc4ee90eec49a8488fa44ccc527aefe226e7be0dde1a73caa54ba7745"}
Oct 29 22:03:12.059986 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:12.059948    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw" event={"ID":"728837f8-b92c-4025-b332-663f29e1570b","Type":"ContainerStarted","Data":"2480f00b1eaa05d888287efc332939fa50f1a71439b07809eef0ddb76e1e5499"}
Oct 29 22:03:12.156602 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:12.156547    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw" podStartSLOduration=1.156533891 podStartE2EDuration="1.156533891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:12.154087792 +0000 UTC m=+1013.227044793" watchObservedRunningTime="2026-10-29 22:03:12.156533891 +0000 UTC m=+1013.229490856"
Oct 29 22:03:12.512033 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:12.512001    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:12.513168 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:12.513143    4962 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:03:12.513321 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:12.513196    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw" podUID="728837f8-b92c-4025-b332-663f29e1570b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 29 22:03:13.511367 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:13.511331    4962 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:03:13.511574 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:13.511389    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw" podUID="728837f8-b92c-4025-b332-663f29e1570b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 29 22:03:13.908466 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:13.908432    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"]
Oct 29 22:03:13.913433 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:13.913403    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:13.939982 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:13.939951    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istiod-ca-cert\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:13.940136 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:13.940006    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-workload-certs\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:13.940136 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:13.940088    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-token\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:13.940136 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:13.940124    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-workload-socket\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:13.940338 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:13.940266    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-podinfo\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:13.940404 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:13.940377    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8msr9\" (UniqueName: \"kubernetes.io/projected/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-kube-api-access-8msr9\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:13.940534 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:13.940518    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-envoy\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:13.940598 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:13.940567    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-credential-socket\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:13.940652 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:13.940622    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-data\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:13.946779 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:13.946752    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"]
Oct 29 22:03:13.954901 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:13.954862    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-6df539-openshift-default-dockercfg-xmnlz\""
Oct 29 22:03:14.066312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:14.066070    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-token\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:14.066312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:14.066137    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-workload-socket\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:14.066312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:14.066301    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-podinfo\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:14.066622 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:14.066396    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8msr9\" (UniqueName: \"kubernetes.io/projected/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-kube-api-access-8msr9\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:14.066622 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:14.066451    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-envoy\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:14.066622 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:14.066512    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-credential-socket\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:14.066622 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:14.066573    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-data\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:14.066833 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:14.066701    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istiod-ca-cert\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:14.066833 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:14.066737    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-workload-certs\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:14.067336 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:14.067303    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-workload-certs\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:14.154715 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:14.067817    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-credential-socket\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:14.154940 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:14.068265    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-workload-socket\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:14.155074 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:14.091699    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-data\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:14.155191 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:14.092463    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istiod-ca-cert\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:14.155363 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:14.152991    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-podinfo\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:14.203492 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:14.166627    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8msr9\" (UniqueName: \"kubernetes.io/projected/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-kube-api-access-8msr9\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:14.285680 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:14.268600    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-envoy\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:14.301724 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:14.301496    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-token\") pod \"e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") " pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:14.512514 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:14.512422    4962 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:03:14.543166 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:14.520020    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw" podUID="728837f8-b92c-4025-b332-663f29e1570b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 29 22:03:14.543493 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:14.540678    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:15.240802 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:15.240771    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"]
Oct 29 22:03:15.241826 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:03:15.241792    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630849f8_61af_40a9_80b9_7f4ae5a4ee6e.slice/crio-d71f8af3f5bfc7362794682bb6c1cf51ad9db43f599b0bb696d46db30795ee55 WatchSource:0}: Error finding container d71f8af3f5bfc7362794682bb6c1cf51ad9db43f599b0bb696d46db30795ee55: Status 404 returned error can't find the container with id d71f8af3f5bfc7362794682bb6c1cf51ad9db43f599b0bb696d46db30795ee55
Oct 29 22:03:15.244771 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:15.244738    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:03:15.244870 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:15.244817    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:03:15.244870 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:15.244851    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:03:15.262045 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:15.262012    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k" event={"ID":"630849f8-61af-40a9-80b9-7f4ae5a4ee6e","Type":"ContainerStarted","Data":"d71f8af3f5bfc7362794682bb6c1cf51ad9db43f599b0bb696d46db30795ee55"}
Oct 29 22:03:15.512595 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:15.512512    4962 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:03:15.512595 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:15.512565    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw" podUID="728837f8-b92c-4025-b332-663f29e1570b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 29 22:03:16.270221 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:16.270175    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k" event={"ID":"630849f8-61af-40a9-80b9-7f4ae5a4ee6e","Type":"ContainerStarted","Data":"52700b1df6d64125547285c51c1e9026f981585a8ffa49ac9fc2ee9e31661c7e"}
Oct 29 22:03:16.351130 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:16.351080    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k" podStartSLOduration=3.351066481 podStartE2EDuration="3.351066481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:16.338972895 +0000 UTC m=+1017.411929859" watchObservedRunningTime="2026-10-29 22:03:16.351066481 +0000 UTC m=+1017.424023445"
Oct 29 22:03:16.520363 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:16.512679    4962 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:03:16.520363 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:16.512752    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw" podUID="728837f8-b92c-4025-b332-663f29e1570b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 29 22:03:16.547365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:16.543679    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:16.584313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:16.573586    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:17.300345 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:17.300321    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:17.301857 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:17.301836    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:03:17.515366 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:17.514826    4962 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:03:17.515366 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:17.514922    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw" podUID="728837f8-b92c-4025-b332-663f29e1570b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 29 22:03:18.511463 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:18.511427    4962 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Oct 29 22:03:18.512002 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:18.511481    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw" podUID="728837f8-b92c-4025-b332-663f29e1570b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Oct 29 22:03:19.515819 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:19.515792    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:19.516312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:19.516277    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:19.516873 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:19.516851    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:03:20.630540 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:20.630500    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx"]
Oct 29 22:03:20.636471 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:20.636448    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx"
Oct 29 22:03:20.660582 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:20.660551    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx"]
Oct 29 22:03:20.666423 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:20.666395    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-6df539-dockercfg-rw7rt\""
Oct 29 22:03:20.683954 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:20.683926    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0173eb5-ad53-4ba5-8abd-6bfe9d27d989-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx\" (UID: \"d0173eb5-ad53-4ba5-8abd-6bfe9d27d989\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx"
Oct 29 22:03:20.684135 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:20.684045    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9tcr\" (UniqueName: \"kubernetes.io/projected/d0173eb5-ad53-4ba5-8abd-6bfe9d27d989-kube-api-access-c9tcr\") pod \"payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx\" (UID: \"d0173eb5-ad53-4ba5-8abd-6bfe9d27d989\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx"
Oct 29 22:03:20.745311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:20.744571    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-6df539\""
Oct 29 22:03:20.806311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:20.804703    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0173eb5-ad53-4ba5-8abd-6bfe9d27d989-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx\" (UID: \"d0173eb5-ad53-4ba5-8abd-6bfe9d27d989\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx"
Oct 29 22:03:20.806311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:20.804800    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9tcr\" (UniqueName: \"kubernetes.io/projected/d0173eb5-ad53-4ba5-8abd-6bfe9d27d989-kube-api-access-c9tcr\") pod \"payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx\" (UID: \"d0173eb5-ad53-4ba5-8abd-6bfe9d27d989\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx"
Oct 29 22:03:20.837310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:20.832443    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0173eb5-ad53-4ba5-8abd-6bfe9d27d989-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx\" (UID: \"d0173eb5-ad53-4ba5-8abd-6bfe9d27d989\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx"
Oct 29 22:03:20.866919 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:20.866886    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9tcr\" (UniqueName: \"kubernetes.io/projected/d0173eb5-ad53-4ba5-8abd-6bfe9d27d989-kube-api-access-c9tcr\") pod \"payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx\" (UID: \"d0173eb5-ad53-4ba5-8abd-6bfe9d27d989\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx"
Oct 29 22:03:20.957311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:20.954381    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx"
Oct 29 22:03:21.424220 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:03:21.424186    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0173eb5_ad53_4ba5_8abd_6bfe9d27d989.slice/crio-27856133ff556a2e71c5c53a1650db53b0d78e5d04e4f88bb9824d304431e40f WatchSource:0}: Error finding container 27856133ff556a2e71c5c53a1650db53b0d78e5d04e4f88bb9824d304431e40f: Status 404 returned error can't find the container with id 27856133ff556a2e71c5c53a1650db53b0d78e5d04e4f88bb9824d304431e40f
Oct 29 22:03:21.425118 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:21.425086    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx"]
Oct 29 22:03:22.402161 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:22.402112    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx" event={"ID":"d0173eb5-ad53-4ba5-8abd-6bfe9d27d989","Type":"ContainerStarted","Data":"ef00db8e5b187649d7b9cf8d32e7d4fc37df16774a77fa71d5482023de3de251"}
Oct 29 22:03:22.402872 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:22.402842    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx" event={"ID":"d0173eb5-ad53-4ba5-8abd-6bfe9d27d989","Type":"ContainerStarted","Data":"27856133ff556a2e71c5c53a1650db53b0d78e5d04e4f88bb9824d304431e40f"}
Oct 29 22:03:22.410621 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:22.404273    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx"
Oct 29 22:03:34.420757 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:34.420679    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx"
Oct 29 22:03:34.467601 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:34.467557    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx" podStartSLOduration=14.467543364 podStartE2EDuration="14.467543364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:22.466626099 +0000 UTC m=+1023.539583064" watchObservedRunningTime="2026-10-29 22:03:34.467543364 +0000 UTC m=+1035.540500385"
Oct 29 22:03:35.645190 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:35.645157    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"]
Oct 29 22:03:35.650236 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:35.650213    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"
Oct 29 22:03:35.674393 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:35.674367    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-6df539-serving-cert\""
Oct 29 22:03:35.680186 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:35.680162    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"]
Oct 29 22:03:35.705650 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:35.705624    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-openshift-service-ca\") pod \"maas-api-e2e-api-b-6df539-7787bbf957-5gjfk\" (UID: \"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"
Oct 29 22:03:35.705806 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:35.705776    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzcm7\" (UniqueName: \"kubernetes.io/projected/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-kube-api-access-lzcm7\") pod \"maas-api-e2e-api-b-6df539-7787bbf957-5gjfk\" (UID: \"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"
Oct 29 22:03:35.705915 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:35.705894    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-metrics-certs\") pod \"maas-api-e2e-api-b-6df539-7787bbf957-5gjfk\" (UID: \"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"
Oct 29 22:03:35.706044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:35.706033    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-maas-api-tls\") pod \"maas-api-e2e-api-b-6df539-7787bbf957-5gjfk\" (UID: \"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"
Oct 29 22:03:35.807796 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:35.807766    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-openshift-service-ca\") pod \"maas-api-e2e-api-b-6df539-7787bbf957-5gjfk\" (UID: \"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"
Oct 29 22:03:35.807925 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:35.807854    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzcm7\" (UniqueName: \"kubernetes.io/projected/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-kube-api-access-lzcm7\") pod \"maas-api-e2e-api-b-6df539-7787bbf957-5gjfk\" (UID: \"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"
Oct 29 22:03:35.807925 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:35.807906    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-metrics-certs\") pod \"maas-api-e2e-api-b-6df539-7787bbf957-5gjfk\" (UID: \"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"
Oct 29 22:03:35.808041 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:35.808028    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-maas-api-tls\") pod \"maas-api-e2e-api-b-6df539-7787bbf957-5gjfk\" (UID: \"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"
Oct 29 22:03:35.808201 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:03:35.808182    4962 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-serving-cert: secret "maas-api-e2e-api-b-6df539-serving-cert" not found
Oct 29 22:03:35.808278 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:03:35.808266    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-maas-api-tls podName:d5e9b803-7f4a-42eb-9535-9cd54d1ae76a nodeName:}" failed. No retries permitted until 2026-10-29 22:03:36.308246439 +0000 UTC m=+1037.381203398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-maas-api-tls") pod "maas-api-e2e-api-b-6df539-7787bbf957-5gjfk" (UID: "d5e9b803-7f4a-42eb-9535-9cd54d1ae76a") : secret "maas-api-e2e-api-b-6df539-serving-cert" not found
Oct 29 22:03:35.808567 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:35.808549    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-openshift-service-ca\") pod \"maas-api-e2e-api-b-6df539-7787bbf957-5gjfk\" (UID: \"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"
Oct 29 22:03:35.815697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:35.810657    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-metrics-certs\") pod \"maas-api-e2e-api-b-6df539-7787bbf957-5gjfk\" (UID: \"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"
Oct 29 22:03:35.873187 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:35.872846    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzcm7\" (UniqueName: \"kubernetes.io/projected/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-kube-api-access-lzcm7\") pod \"maas-api-e2e-api-b-6df539-7787bbf957-5gjfk\" (UID: \"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"
Oct 29 22:03:36.316280 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:36.316247    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-maas-api-tls\") pod \"maas-api-e2e-api-b-6df539-7787bbf957-5gjfk\" (UID: \"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"
Oct 29 22:03:36.318791 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:36.318762    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-maas-api-tls\") pod \"maas-api-e2e-api-b-6df539-7787bbf957-5gjfk\" (UID: \"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"
Oct 29 22:03:36.568375 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:36.567624    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"
Oct 29 22:03:37.018942 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:37.018899    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"]
Oct 29 22:03:37.679846 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:37.678786    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk" event={"ID":"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a","Type":"ContainerStarted","Data":"ef7ea00647a275fbbd8e09fd87523f42581ef3e803a7f90b8cfb2740a74ce381"}
Oct 29 22:03:37.679846 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:37.678854    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk" event={"ID":"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a","Type":"ContainerStarted","Data":"a67f282611fe889b7ddf6fa111f9c9a78132cd9e025db83188d11533fa6f45d3"}
Oct 29 22:03:37.681614 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:37.681530    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"
Oct 29 22:03:37.735026 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:37.734967    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk" podStartSLOduration=2.256713891 podStartE2EDuration="2.734952318s" totalImagesPullingTime="478.238427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:37.731543988 +0000 UTC m=+1038.804500994" watchObservedRunningTime="2026-10-29 22:03:37.734952318 +0000 UTC m=+1038.807909281"
Oct 29 22:03:39.721890 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:03:39.721854    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:03:41.280196 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.280162    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"]
Oct 29 22:03:41.284978 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.284958    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.311331 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.311272    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"]
Oct 29 22:03:41.414044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.414010    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2n82\" (UniqueName: \"kubernetes.io/projected/159da2ec-dc40-466b-ad09-0061dbd852a1-kube-api-access-t2n82\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.414211 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.414062    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-token\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.414211 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.414144    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-workload-certs\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.414211 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.414206    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-envoy\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.414366 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.414224    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159da2ec-dc40-466b-ad09-0061dbd852a1-istiod-ca-cert\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.414416 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.414390    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-data\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.414454 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.414422    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-workload-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.414485 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.414448    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-podinfo\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.414550 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.414534    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-credential-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.516413 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.516377    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2n82\" (UniqueName: \"kubernetes.io/projected/159da2ec-dc40-466b-ad09-0061dbd852a1-kube-api-access-t2n82\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.516563 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.516432    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-token\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.516608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.516584    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-workload-certs\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.516761 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.516742    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-envoy\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.516801 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.516777    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159da2ec-dc40-466b-ad09-0061dbd852a1-istiod-ca-cert\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.516863 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.516852    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-data\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.516910 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.516880    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-workload-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.516945 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.516922    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-podinfo\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.516992 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.516945    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-workload-certs\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.517041 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.517025    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-credential-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.517198 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.517177    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-data\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.517276 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.517246    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-workload-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.517433 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.517412    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-credential-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.517631 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.517610    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159da2ec-dc40-466b-ad09-0061dbd852a1-istiod-ca-cert\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.519041 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.519022    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-envoy\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.519362 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.519345    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-podinfo\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.539321 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.539223    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-token\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.539487 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.539468    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2n82\" (UniqueName: \"kubernetes.io/projected/159da2ec-dc40-466b-ad09-0061dbd852a1-kube-api-access-t2n82\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.599320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.599282    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:41.936160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.936106    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:03:41.936386 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.936248    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:03:41.936459 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.936418    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:03:41.944783 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:41.944758    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"]
Oct 29 22:03:42.715336 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:42.715281    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf" event={"ID":"159da2ec-dc40-466b-ad09-0061dbd852a1","Type":"ContainerStarted","Data":"2fdaf48668c0a9160323b582044ad15e29c4079957e189b1c63115847eaefb49"}
Oct 29 22:03:42.715798 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:42.715355    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf" event={"ID":"159da2ec-dc40-466b-ad09-0061dbd852a1","Type":"ContainerStarted","Data":"30e143c9a253848df4a6749d118acb689ac04adac496633d61387c561ef4d911"}
Oct 29 22:03:42.808427 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:42.808370    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf" podStartSLOduration=1.808356334 podStartE2EDuration="1.808356334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:03:42.795815281 +0000 UTC m=+1043.868772246" watchObservedRunningTime="2026-10-29 22:03:42.808356334 +0000 UTC m=+1043.881313297"
Oct 29 22:03:43.599658 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:43.599615    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:43.601115 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:43.601086    4962 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:03:43.601281 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:43.601139    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf" podUID="159da2ec-dc40-466b-ad09-0061dbd852a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 29 22:03:44.599888 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:44.599851    4962 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:03:44.600390 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:44.599912    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf" podUID="159da2ec-dc40-466b-ad09-0061dbd852a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 29 22:03:44.700165 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:44.700137    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"
Oct 29 22:03:45.599812 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:45.599780    4962 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:03:45.600000 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:45.599844    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf" podUID="159da2ec-dc40-466b-ad09-0061dbd852a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 29 22:03:46.600612 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:46.600580    4962 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:03:46.601017 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:46.600631    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf" podUID="159da2ec-dc40-466b-ad09-0061dbd852a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 29 22:03:47.609315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:47.604074    4962 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 29 22:03:47.609315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:47.604149    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf" podUID="159da2ec-dc40-466b-ad09-0061dbd852a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 29 22:03:48.710551 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:48.710518    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:48.712191 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:48.712166    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:03:48.712842 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:03:48.712821    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:04:09.721153 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:04:09.721116    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:04:11.776349 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:11.776320    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7"]
Oct 29 22:04:11.806968 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:11.806942    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt"]
Oct 29 22:04:16.777460 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:16.777413    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7" podUID="c01b8e41-5665-45da-85ff-8ca4b229b30e" containerName="payload-processing" containerID="cri-o://2f0f07096a30bd3c685f4a13454618caf51b3eb22bf3e4398eadd67a55acb233" gracePeriod=25
Oct 29 22:04:16.808134 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:16.808089    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt" podUID="f74f6919-d222-4c09-8b7d-6a1aa56229a3" containerName="payload-pre-processing" containerID="cri-o://3abcdff572f061b07a70368e1713b96bc4019bcb0c59a79bb192b7dff0de83ac" gracePeriod=25
Oct 29 22:04:17.172633 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:17.172091    4962 generic.go:356] "Generic (PLEG): container finished" podID="c01b8e41-5665-45da-85ff-8ca4b229b30e" containerID="2f0f07096a30bd3c685f4a13454618caf51b3eb22bf3e4398eadd67a55acb233" exitCode=0
Oct 29 22:04:17.172633 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:17.172193    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7" event={"ID":"c01b8e41-5665-45da-85ff-8ca4b229b30e","Type":"ContainerDied","Data":"2f0f07096a30bd3c685f4a13454618caf51b3eb22bf3e4398eadd67a55acb233"}
Oct 29 22:04:17.176346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:17.174388    4962 generic.go:356] "Generic (PLEG): container finished" podID="f74f6919-d222-4c09-8b7d-6a1aa56229a3" containerID="3abcdff572f061b07a70368e1713b96bc4019bcb0c59a79bb192b7dff0de83ac" exitCode=0
Oct 29 22:04:17.176346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:17.174590    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt" event={"ID":"f74f6919-d222-4c09-8b7d-6a1aa56229a3","Type":"ContainerDied","Data":"3abcdff572f061b07a70368e1713b96bc4019bcb0c59a79bb192b7dff0de83ac"}
Oct 29 22:04:17.265256 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:17.265222    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7"
Oct 29 22:04:17.345668 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:17.345635    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt"
Oct 29 22:04:17.346774 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:17.346744    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c01b8e41-5665-45da-85ff-8ca4b229b30e-kube-api-access-km92m\" (UniqueName: \"kubernetes.io/projected/c01b8e41-5665-45da-85ff-8ca4b229b30e-kube-api-access-km92m\") pod \"c01b8e41-5665-45da-85ff-8ca4b229b30e\" (UID: \"c01b8e41-5665-45da-85ff-8ca4b229b30e\") "
Oct 29 22:04:17.346922 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:17.346799    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c01b8e41-5665-45da-85ff-8ca4b229b30e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c01b8e41-5665-45da-85ff-8ca4b229b30e-plugins-config-volume\") pod \"c01b8e41-5665-45da-85ff-8ca4b229b30e\" (UID: \"c01b8e41-5665-45da-85ff-8ca4b229b30e\") "
Oct 29 22:04:17.347206 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:17.347171    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c01b8e41-5665-45da-85ff-8ca4b229b30e-plugins-config-volume" pod "c01b8e41-5665-45da-85ff-8ca4b229b30e" (UID: "c01b8e41-5665-45da-85ff-8ca4b229b30e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:04:17.347603 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:17.347583    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c01b8e41-5665-45da-85ff-8ca4b229b30e-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:04:17.349303 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:17.349262    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01b8e41-5665-45da-85ff-8ca4b229b30e-kube-api-access-km92m" pod "c01b8e41-5665-45da-85ff-8ca4b229b30e" (UID: "c01b8e41-5665-45da-85ff-8ca4b229b30e"). InnerVolumeSpecName "kube-api-access-km92m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:04:17.448940 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:17.448904    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f74f6919-d222-4c09-8b7d-6a1aa56229a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f74f6919-d222-4c09-8b7d-6a1aa56229a3-plugins-config-volume\") pod \"f74f6919-d222-4c09-8b7d-6a1aa56229a3\" (UID: \"f74f6919-d222-4c09-8b7d-6a1aa56229a3\") "
Oct 29 22:04:17.449166 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:17.449149    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f74f6919-d222-4c09-8b7d-6a1aa56229a3-kube-api-access-c6gbs\" (UniqueName: \"kubernetes.io/projected/f74f6919-d222-4c09-8b7d-6a1aa56229a3-kube-api-access-c6gbs\") pod \"f74f6919-d222-4c09-8b7d-6a1aa56229a3\" (UID: \"f74f6919-d222-4c09-8b7d-6a1aa56229a3\") "
Oct 29 22:04:17.449378 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:17.449352    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f74f6919-d222-4c09-8b7d-6a1aa56229a3-plugins-config-volume" pod "f74f6919-d222-4c09-8b7d-6a1aa56229a3" (UID: "f74f6919-d222-4c09-8b7d-6a1aa56229a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:04:17.449913 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:17.449889    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km92m\" (UniqueName: \"kubernetes.io/projected/c01b8e41-5665-45da-85ff-8ca4b229b30e-kube-api-access-km92m\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:04:17.450035 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:17.449919    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f74f6919-d222-4c09-8b7d-6a1aa56229a3-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:04:17.451324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:17.451268    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74f6919-d222-4c09-8b7d-6a1aa56229a3-kube-api-access-c6gbs" pod "f74f6919-d222-4c09-8b7d-6a1aa56229a3" (UID: "f74f6919-d222-4c09-8b7d-6a1aa56229a3"). InnerVolumeSpecName "kube-api-access-c6gbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:04:17.551300 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:17.551264    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6gbs\" (UniqueName: \"kubernetes.io/projected/f74f6919-d222-4c09-8b7d-6a1aa56229a3-kube-api-access-c6gbs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:04:18.185079 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:18.185048    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7"
Oct 29 22:04:18.185550 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:18.185059    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7" event={"ID":"c01b8e41-5665-45da-85ff-8ca4b229b30e","Type":"ContainerDied","Data":"eb0aa4919974588b579dc5676feaed1f415275a389170649cfa8f2f0c963f5c1"}
Oct 29 22:04:18.185550 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:18.185174    4962 scope.go:122] "RemoveContainer" containerID="2f0f07096a30bd3c685f4a13454618caf51b3eb22bf3e4398eadd67a55acb233"
Oct 29 22:04:18.186538 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:18.186496    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt" event={"ID":"f74f6919-d222-4c09-8b7d-6a1aa56229a3","Type":"ContainerDied","Data":"1b37ea6733f6dba285034dcadc63119c4504da812528c49d01408977ce54ca42"}
Oct 29 22:04:18.186619 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:18.186538    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt"
Oct 29 22:04:18.197096 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:18.197063    4962 scope.go:122] "RemoveContainer" containerID="3abcdff572f061b07a70368e1713b96bc4019bcb0c59a79bb192b7dff0de83ac"
Oct 29 22:04:18.237622 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:18.237600    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7"]
Oct 29 22:04:18.249342 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:18.249280    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a9f9aa-6697b44f6d-p69w7"]
Oct 29 22:04:18.283725 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:18.283702    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt"]
Oct 29 22:04:18.288760 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:18.288738    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a9f9aa-7479bf578c-wzvzt"]
Oct 29 22:04:19.070610 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:19.070576    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c01b8e41-5665-45da-85ff-8ca4b229b30e" path="/var/lib/kubelet/pods/c01b8e41-5665-45da-85ff-8ca4b229b30e/volumes"
Oct 29 22:04:19.071083 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:19.071060    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f74f6919-d222-4c09-8b7d-6a1aa56229a3" path="/var/lib/kubelet/pods/f74f6919-d222-4c09-8b7d-6a1aa56229a3/volumes"
Oct 29 22:04:24.206304 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:24.205947    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"]
Oct 29 22:04:24.207313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:24.207148    4962 cpu_manager.go:414] "containerMap: removing container" podUID="c01b8e41-5665-45da-85ff-8ca4b229b30e" containerName="payload-processing"
Oct 29 22:04:24.207313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:24.207174    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c01b8e41-5665-45da-85ff-8ca4b229b30e" containerName="payload-processing"
Oct 29 22:04:24.207313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:24.207193    4962 cpu_manager.go:414] "containerMap: removing container" podUID="f74f6919-d222-4c09-8b7d-6a1aa56229a3" containerName="payload-pre-processing"
Oct 29 22:04:24.207313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:24.207202    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f74f6919-d222-4c09-8b7d-6a1aa56229a3" containerName="payload-pre-processing"
Oct 29 22:04:24.207602 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:24.207403    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="c01b8e41-5665-45da-85ff-8ca4b229b30e" containerName="payload-processing"
Oct 29 22:04:24.207602 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:24.207418    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="f74f6919-d222-4c09-8b7d-6a1aa56229a3" containerName="payload-pre-processing"
Oct 29 22:04:24.213817 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:24.213795    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"
Oct 29 22:04:24.313311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:24.288154    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-734480-serving-cert\""
Oct 29 22:04:24.343163 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:24.342354    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"]
Oct 29 22:04:24.438310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:24.436415    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6\" (UID: \"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"
Oct 29 22:04:24.438310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:24.436506    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6\" (UID: \"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"
Oct 29 22:04:24.438310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:24.436572    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9j6t\" (UniqueName: \"kubernetes.io/projected/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-kube-api-access-c9j6t\") pod \"maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6\" (UID: \"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"
Oct 29 22:04:24.438310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:24.436835    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-metrics-certs\") pod \"maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6\" (UID: \"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"
Oct 29 22:04:24.546596 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:24.546504    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-metrics-certs\") pod \"maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6\" (UID: \"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"
Oct 29 22:04:24.546868 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:24.546839    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6\" (UID: \"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"
Oct 29 22:04:24.547061 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:24.547047    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6\" (UID: \"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"
Oct 29 22:04:24.547227 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:24.547208    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9j6t\" (UniqueName: \"kubernetes.io/projected/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-kube-api-access-c9j6t\") pod \"maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6\" (UID: \"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"
Oct 29 22:04:24.578314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:24.557190    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6\" (UID: \"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"
Oct 29 22:04:24.578314 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:04:24.557342    4962 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-serving-cert: secret "maas-api-e2e-ipp-a-734480-serving-cert" not found
Oct 29 22:04:24.578314 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:04:24.557415    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-maas-api-tls podName:ed7fcffb-6974-4e95-b689-bc8eb07d9dc7 nodeName:}" failed. No retries permitted until 2026-10-29 22:04:25.057396148 +0000 UTC m=+1086.130353097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-maas-api-tls") pod "maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6" (UID: "ed7fcffb-6974-4e95-b689-bc8eb07d9dc7") : secret "maas-api-e2e-ipp-a-734480-serving-cert" not found
Oct 29 22:04:24.809597 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:24.809502    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:24.850874 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:24.850834    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9j6t\" (UniqueName: \"kubernetes.io/projected/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-kube-api-access-c9j6t\") pod \"maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6\" (UID: \"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"
Oct 29 22:04:24.891802 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:24.852511    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:24.980651 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:24.980602    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:25.036309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:25.033459    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:25.058808 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:25.058772    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6\" (UID: \"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"
Oct 29 22:04:25.070375 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:04:25.059091    4962 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-serving-cert: secret "maas-api-e2e-ipp-a-734480-serving-cert" not found
Oct 29 22:04:25.070778 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:04:25.070751    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-maas-api-tls podName:ed7fcffb-6974-4e95-b689-bc8eb07d9dc7 nodeName:}" failed. No retries permitted until 2026-10-29 22:04:26.070719115 +0000 UTC m=+1087.143676070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-maas-api-tls") pod "maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6" (UID: "ed7fcffb-6974-4e95-b689-bc8eb07d9dc7") : secret "maas-api-e2e-ipp-a-734480-serving-cert" not found
Oct 29 22:04:25.127652 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:25.119078    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-metrics-certs\") pod \"maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6\" (UID: \"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"
Oct 29 22:04:25.157751 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:25.157716    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:25.157979 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:25.157955    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr" podUID="4ef87fe2-429a-4393-8ddf-fa43e2dd9044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:25.170393 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:25.170349    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:25.256300 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:25.235277    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:25.439441 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:25.439389    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:25.488923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:25.488866    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:04:25.558430 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:25.558379    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:04:25.824396 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:25.758583    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:25.824396 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:25.810363    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:04:25.824396 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:25.810948    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:04:25.921312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:25.905385    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:04:25.921312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:25.905967    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 22:04:26.071314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:26.065368    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:26.071314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:26.065944    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:04:26.106412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:26.106314    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6\" (UID: \"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"
Oct 29 22:04:26.107031 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:04:26.107004    4962 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-serving-cert: secret "maas-api-e2e-ipp-a-734480-serving-cert" not found
Oct 29 22:04:26.107216 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:04:26.107202    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-maas-api-tls podName:ed7fcffb-6974-4e95-b689-bc8eb07d9dc7 nodeName:}" failed. No retries permitted until 2026-10-29 22:04:28.107179508 +0000 UTC m=+1089.180136454 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-maas-api-tls") pod "maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6" (UID: "ed7fcffb-6974-4e95-b689-bc8eb07d9dc7") : secret "maas-api-e2e-ipp-a-734480-serving-cert" not found
Oct 29 22:04:26.182317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:26.176508    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:26.251610 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:26.242833    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:26.280760 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:26.280727    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:04:26.292447 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:26.292412    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:04:26.370498 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:26.370396    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:26.467314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:26.465447    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:04:26.475346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:26.473446    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 22:04:26.513312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:26.509582    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:26.548580 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:26.548551    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 22:04:26.599006 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:26.582405    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:26.606662 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:26.606635    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 22:04:26.644707 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:26.582449    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:26.645009 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:26.644974    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5" podUID="9762d8d0-a759-42ac-bc1b-c948adfa6f9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:26.897511 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:26.897378    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:04:26.986313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:26.984916    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:04:27.016315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:27.003917    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:04:27.492313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:27.466318    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:27.492313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:27.489481    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:27.544372 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:27.533314    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:27.551558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:27.550432    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:04:27.577652 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:27.562892    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:27.605735 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:27.605690    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:27.654962 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:27.635985    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:04:27.745904 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:27.745812    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:28.129733 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:28.129627    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw" podUID="f6631eff-5d71-483c-aac7-8d578274ce76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:28.129971 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:27.926516    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:28.130081 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:28.065824    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:28.135976 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:28.065926    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:28.195312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:27.926554    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:28.298567 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:28.287895    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6\" (UID: \"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"
Oct 29 22:04:28.468561 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:28.468504    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:28.492533 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:28.492490    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:28.550247 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:28.550196    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:28.564687 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:28.564653    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:04:28.641281 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:28.641235    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:04:28.641994 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:28.640388    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:04:28.658660 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:28.658617    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:28.673593 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:28.673557    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6\" (UID: \"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"
Oct 29 22:04:28.697778 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:04:28.697743    4962 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.508s"
Oct 29 22:04:28.699366 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:28.699342    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:04:28.699705 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:28.699692    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 22:04:28.739105 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:28.739022    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"
Oct 29 22:04:29.254677 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:29.254378    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:04:29.282372 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:29.256639    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": context deadline exceeded"
Oct 29 22:04:29.282654 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:29.272871    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:04:29.383665 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:29.383559    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:29.499863 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:29.499818    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:04:29.519647 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:29.502242    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:29.561311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:29.554034    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:29.649237 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:29.621269    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:29.672276 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:29.671784    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:29.672276 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:29.672170    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:04:29.859415 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:29.814471    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:29.977308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:29.836911    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:29.977308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:29.971027    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr" podUID="10c5ec48-f48a-4703-a82c-9a6995506125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.313949 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:30.313842    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:30.314337 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:30.314206    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6" podUID="64a76b6c-1542-4fb1-9730-7b770c452b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.320318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:30.319320    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:04:30.322305 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:30.320530    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:04:30.453318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:30.429831    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:04:30.486310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:30.440553    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.541888 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:30.541843    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:04:30.560967 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:30.560907    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.562782 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:30.562747    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.676390 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:30.667143    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.676390 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:30.672366    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:30.676390 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:30.672695    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:04:31.040465 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:31.037032    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": context deadline exceeded"
Oct 29 22:04:31.040465 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:31.037139    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:04:31.067987 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:31.054322    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:31.372164 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:31.348137    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:04:31.377333 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:31.374176    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:04:31.493309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:31.463300    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:31.710495 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:31.710441    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:04:31.737408 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:31.732384    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:04:31.778734 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:31.778671    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:31.780541 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:31.780434    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:04:31.780950 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:31.780907    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:04:31.782344 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:31.782315    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:04:32.114514 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:32.073262    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded"
Oct 29 22:04:32.114514 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:32.098384    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:32.303313 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:04:32.247915    4962 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.195s"
Oct 29 22:04:32.346283 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:32.346251    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"]
Oct 29 22:04:32.475315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:32.286421    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:32.475315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:32.456736    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z" podUID="132ed775-c9e6-41f1-b224-fff550c6e073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:32.475315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:32.386606    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:32.475315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:32.373464    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:32.708771 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:32.708732    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:04:32.759473 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:32.718646    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:32.780849 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:32.740748    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:32.780849 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:32.741822    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded"
Oct 29 22:04:32.833110 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:32.832778    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:32.833977 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:32.833931    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:04:32.834507 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:32.834485    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:04:32.834978 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:32.834958    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:04:32.923901 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:32.923863    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:32.924162 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:32.924137    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv" podUID="d71de0ec-1eaa-481f-8cee-cb256e330136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:33.104366 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:33.101469    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:33.255870 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:33.255772    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:33.256224 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:33.256191    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2" podUID="03b2ca39-a1c7-45bf-8458-f23215d334b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:33.595887 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:33.595775    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:33.596651 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:33.596612    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:04:33.760618 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:33.760575    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:33.871174 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:33.836371    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:04:33.871174 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:33.837595    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:33.871174 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:33.837917    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:04:33.871174 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:33.838202    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:04:33.871174 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:33.852490    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6" event={"ID":"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7","Type":"ContainerStarted","Data":"9cd04fde088f8d7e43690ee4d7d8f1277cf95e1e998b165feee440809bcd417e"}
Oct 29 22:04:33.959323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:33.957346    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.026310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:33.975390    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.104077 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:34.103957    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:04:34.212675 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:34.212401    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:34.212675 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:34.212486    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc" podUID="35c2b463-e2a6-4b1d-8425-558ca7832668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.221447 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:34.221412    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:34.221657 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:34.221629    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc" podUID="036f71f6-61b8-469d-a080-d1ab0b45c195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.263464 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:34.263417    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.571754 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:34.560492    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.605754 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:34.600401    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.605754 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:34.600812    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.605754 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:34.601399    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:04:34.605754 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:34.601441    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42" podUID="e1897161-0016-400a-9928-8b407f4148e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.767352 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:34.763957    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.839434 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:34.839386    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:34.839833 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:34.839788    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:04:34.840128 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:34.840089    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:04:34.840406 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:34.840380    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:04:34.962312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:34.961372    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:35.039369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:35.034382    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:04:35.138318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:35.137596    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:35.517575 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:35.517534    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6" event={"ID":"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7","Type":"ContainerStarted","Data":"135a7912771f066cceb519f4c7f0eac1cf5a0fd2359c9f3f7eb1a27239cadaf5"}
Oct 29 22:04:35.521772 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:35.521744    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"
Oct 29 22:04:35.601494 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:35.601456    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:04:35.602068 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:35.602037    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:04:35.745657 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:35.745402    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:35.780686 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:35.779634    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:04:36.030777 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:36.030665    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:36.038300 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:36.031063    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:36.038300 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:36.031375    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:36.038300 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:36.031632    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:36.081905 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:36.079620    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:04:36.180332 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:36.179495    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:36.216360 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:36.206478    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:36.647337 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:36.606464    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:04:36.647337 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:36.608617    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:04:36.788313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:36.785851    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:37.052464 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:37.052414    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:37.053235 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:37.053205    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:04:37.053611 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:37.053587    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:04:37.054011 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:37.053987    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:04:37.109342 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:37.109279    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:04:37.212980 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:37.181064    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:04:37.212980 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:37.212231    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:37.665316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:37.613060    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:04:37.697316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:37.613370    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:37.849316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:37.820816    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:04:37.912478 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:37.912430    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:38.074379 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:38.054373    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:38.074379 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:38.054735    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:04:38.074379 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:38.055008    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:04:38.074379 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:38.055255    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:04:38.229310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:38.225541    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:38.229310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:38.225946    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:04:38.229310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:38.229011    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:04:38.718874 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:38.712521    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:04:38.743329 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:38.719155    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:38.834791 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:38.834744    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:39.072193 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:39.070382    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:39.118340 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:39.116500    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:39.173217 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:39.154199    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:39.173217 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:39.154536    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:04:39.173217 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:39.154778    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:04:39.282311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:39.259280    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:39.282768 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:39.264672    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:04:39.322505 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:39.310555    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:04:39.522886 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:39.522856    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 22:04:39.523550 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:39.523527    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 22:04:39.570471 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:39.570444    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:04:39.584460 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:39.584393    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 22:04:39.613369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:39.613314    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6" podStartSLOduration=14.064849023 podStartE2EDuration="15.613280594s" totalImagesPullingTime="1.548431571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:35.58668539 +0000 UTC m=+1096.659642358" watchObservedRunningTime="2026-10-29 22:04:39.613280594 +0000 UTC m=+1100.686237559"
Oct 29 22:04:39.675154 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:39.674076    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:04:39.705418 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:39.705378    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:04:39.716489 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:39.716112    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:04:39.725451 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:39.725433    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 22:04:39.753355 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:04:39.753271    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:04:39.762546 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:39.762527    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:04:39.798814 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:39.798791    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:04:39.815757 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:39.815737    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 22:04:41.908344 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:41.908255    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"
Oct 29 22:04:43.384067 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:43.384035    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c"]
Oct 29 22:04:43.389027 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:43.389004    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c"
Oct 29 22:04:43.392861 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:43.392838    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksk95\" (UniqueName: \"kubernetes.io/projected/263a0c66-6465-4269-b15c-28ccb8afbe32-kube-api-access-ksk95\") pod \"payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c\" (UID: \"263a0c66-6465-4269-b15c-28ccb8afbe32\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c"
Oct 29 22:04:43.393089 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:43.393068    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263a0c66-6465-4269-b15c-28ccb8afbe32-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c\" (UID: \"263a0c66-6465-4269-b15c-28ccb8afbe32\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c"
Oct 29 22:04:43.408863 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:43.408837    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f6171d\""
Oct 29 22:04:43.410637 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:43.410619    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c"]
Oct 29 22:04:43.417228 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:43.417208    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f6171d-dockercfg-gkjqc\""
Oct 29 22:04:43.507724 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:43.506086    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksk95\" (UniqueName: \"kubernetes.io/projected/263a0c66-6465-4269-b15c-28ccb8afbe32-kube-api-access-ksk95\") pod \"payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c\" (UID: \"263a0c66-6465-4269-b15c-28ccb8afbe32\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c"
Oct 29 22:04:43.507724 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:43.506331    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263a0c66-6465-4269-b15c-28ccb8afbe32-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c\" (UID: \"263a0c66-6465-4269-b15c-28ccb8afbe32\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c"
Oct 29 22:04:43.521322 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:43.507270    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263a0c66-6465-4269-b15c-28ccb8afbe32-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c\" (UID: \"263a0c66-6465-4269-b15c-28ccb8afbe32\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c"
Oct 29 22:04:43.604998 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:43.604912    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksk95\" (UniqueName: \"kubernetes.io/projected/263a0c66-6465-4269-b15c-28ccb8afbe32-kube-api-access-ksk95\") pod \"payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c\" (UID: \"263a0c66-6465-4269-b15c-28ccb8afbe32\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c"
Oct 29 22:04:43.642365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:43.605442    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw"]
Oct 29 22:04:43.712490 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:43.704792    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c"
Oct 29 22:04:43.902642 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:43.902585    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw"
Oct 29 22:04:43.905890 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:43.905864    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw"]
Oct 29 22:04:43.969082 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:43.968703    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d397882-c29a-47e6-a6d5-280f529d89d6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw\" (UID: \"2d397882-c29a-47e6-a6d5-280f529d89d6\") " pod="openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw"
Oct 29 22:04:43.969208 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:43.969094    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2sk4\" (UniqueName: \"kubernetes.io/projected/2d397882-c29a-47e6-a6d5-280f529d89d6-kube-api-access-v2sk4\") pod \"payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw\" (UID: \"2d397882-c29a-47e6-a6d5-280f529d89d6\") " pod="openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw"
Oct 29 22:04:44.071557 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:44.071517    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2sk4\" (UniqueName: \"kubernetes.io/projected/2d397882-c29a-47e6-a6d5-280f529d89d6-kube-api-access-v2sk4\") pod \"payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw\" (UID: \"2d397882-c29a-47e6-a6d5-280f529d89d6\") " pod="openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw"
Oct 29 22:04:44.071742 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:44.071626    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d397882-c29a-47e6-a6d5-280f529d89d6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw\" (UID: \"2d397882-c29a-47e6-a6d5-280f529d89d6\") " pod="openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw"
Oct 29 22:04:44.072443 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:44.072418    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d397882-c29a-47e6-a6d5-280f529d89d6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw\" (UID: \"2d397882-c29a-47e6-a6d5-280f529d89d6\") " pod="openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw"
Oct 29 22:04:44.095192 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:44.095169    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2sk4\" (UniqueName: \"kubernetes.io/projected/2d397882-c29a-47e6-a6d5-280f529d89d6-kube-api-access-v2sk4\") pod \"payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw\" (UID: \"2d397882-c29a-47e6-a6d5-280f529d89d6\") " pod="openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw"
Oct 29 22:04:44.225959 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:04:44.225930    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263a0c66_6465_4269_b15c_28ccb8afbe32.slice/crio-0f41381aa652d9d810994594bb75f088c46dae6b9d64cf532c9f5fa62d6b716d WatchSource:0}: Error finding container 0f41381aa652d9d810994594bb75f088c46dae6b9d64cf532c9f5fa62d6b716d: Status 404 returned error can't find the container with id 0f41381aa652d9d810994594bb75f088c46dae6b9d64cf532c9f5fa62d6b716d
Oct 29 22:04:44.226448 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:44.226424    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c"]
Oct 29 22:04:44.226995 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:44.226976    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw"
Oct 29 22:04:44.554738 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:04:44.554699    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d397882_c29a_47e6_a6d5_280f529d89d6.slice/crio-3a797f17a9669a5b41770d14f76e46b3feb843c0607ae18e2cddddac94f8d190 WatchSource:0}: Error finding container 3a797f17a9669a5b41770d14f76e46b3feb843c0607ae18e2cddddac94f8d190: Status 404 returned error can't find the container with id 3a797f17a9669a5b41770d14f76e46b3feb843c0607ae18e2cddddac94f8d190
Oct 29 22:04:44.556263 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:44.556236    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw"]
Oct 29 22:04:45.111472 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:45.111433    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c" event={"ID":"263a0c66-6465-4269-b15c-28ccb8afbe32","Type":"ContainerStarted","Data":"6d358f03a7fd07efe18ceb64b79c3b1677ae19dd4b5043b9f5332fc5f2dc1533"}
Oct 29 22:04:45.111631 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:45.111490    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c" event={"ID":"263a0c66-6465-4269-b15c-28ccb8afbe32","Type":"ContainerStarted","Data":"0f41381aa652d9d810994594bb75f088c46dae6b9d64cf532c9f5fa62d6b716d"}
Oct 29 22:04:45.111631 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:45.111621    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c"
Oct 29 22:04:45.113321 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:45.113278    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw" event={"ID":"2d397882-c29a-47e6-a6d5-280f529d89d6","Type":"ContainerStarted","Data":"cd9c4448c006a9ad263a8214bb9014d1e12c3322403257fa5d443ceebc769f95"}
Oct 29 22:04:45.113441 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:45.113336    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw" event={"ID":"2d397882-c29a-47e6-a6d5-280f529d89d6","Type":"ContainerStarted","Data":"3a797f17a9669a5b41770d14f76e46b3feb843c0607ae18e2cddddac94f8d190"}
Oct 29 22:04:45.113441 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:45.113362    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw"
Oct 29 22:04:45.158736 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:45.158694    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c" podStartSLOduration=2.158683369 podStartE2EDuration="2.158683369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:45.157532767 +0000 UTC m=+1106.230489732" watchObservedRunningTime="2026-10-29 22:04:45.158683369 +0000 UTC m=+1106.231640334"
Oct 29 22:04:45.232341 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:45.227113    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw" podStartSLOduration=2.227093167 podStartE2EDuration="2.227093167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:45.209997376 +0000 UTC m=+1106.282954450" watchObservedRunningTime="2026-10-29 22:04:45.227093167 +0000 UTC m=+1106.300050131"
Oct 29 22:04:49.596877 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:49.596845    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4"]
Oct 29 22:04:49.602565 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:49.602546    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4"
Oct 29 22:04:49.628218 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:49.628190    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4"]
Oct 29 22:04:49.633750 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:49.633716    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sw7m\" (UniqueName: \"kubernetes.io/projected/37af084a-d949-4a30-a867-54e3cc3943b9-kube-api-access-5sw7m\") pod \"payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4\" (UID: \"37af084a-d949-4a30-a867-54e3cc3943b9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4"
Oct 29 22:04:49.633955 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:49.633932    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37af084a-d949-4a30-a867-54e3cc3943b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4\" (UID: \"37af084a-d949-4a30-a867-54e3cc3943b9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4"
Oct 29 22:04:49.634894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:49.634865    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-380eea\""
Oct 29 22:04:49.634999 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:49.634978    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-380eea-dockercfg-5l2j8\""
Oct 29 22:04:49.851364 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:49.851070    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sw7m\" (UniqueName: \"kubernetes.io/projected/37af084a-d949-4a30-a867-54e3cc3943b9-kube-api-access-5sw7m\") pod \"payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4\" (UID: \"37af084a-d949-4a30-a867-54e3cc3943b9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4"
Oct 29 22:04:49.893518 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:49.882785    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37af084a-d949-4a30-a867-54e3cc3943b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4\" (UID: \"37af084a-d949-4a30-a867-54e3cc3943b9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4"
Oct 29 22:04:49.893518 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:49.851246    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37af084a-d949-4a30-a867-54e3cc3943b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4\" (UID: \"37af084a-d949-4a30-a867-54e3cc3943b9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4"
Oct 29 22:04:50.060368 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:50.010617    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sw7m\" (UniqueName: \"kubernetes.io/projected/37af084a-d949-4a30-a867-54e3cc3943b9-kube-api-access-5sw7m\") pod \"payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4\" (UID: \"37af084a-d949-4a30-a867-54e3cc3943b9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4"
Oct 29 22:04:50.062983 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:50.062158    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9"]
Oct 29 22:04:50.112558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:50.112478    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9"]
Oct 29 22:04:50.112729 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:50.112591    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9"
Oct 29 22:04:50.191689 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:50.191652    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmgmq\" (UniqueName: \"kubernetes.io/projected/f30ae4be-1795-4728-b8b6-075038cff8ff-kube-api-access-rmgmq\") pod \"payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9\" (UID: \"f30ae4be-1795-4728-b8b6-075038cff8ff\") " pod="openshift-ingress/payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9"
Oct 29 22:04:50.191929 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:50.191905    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30ae4be-1795-4728-b8b6-075038cff8ff-plugins-config-volume\") pod \"payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9\" (UID: \"f30ae4be-1795-4728-b8b6-075038cff8ff\") " pod="openshift-ingress/payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9"
Oct 29 22:04:50.217140 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:50.217111    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4"
Oct 29 22:04:50.294114 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:50.294076    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30ae4be-1795-4728-b8b6-075038cff8ff-plugins-config-volume\") pod \"payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9\" (UID: \"f30ae4be-1795-4728-b8b6-075038cff8ff\") " pod="openshift-ingress/payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9"
Oct 29 22:04:50.294349 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:50.294324    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmgmq\" (UniqueName: \"kubernetes.io/projected/f30ae4be-1795-4728-b8b6-075038cff8ff-kube-api-access-rmgmq\") pod \"payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9\" (UID: \"f30ae4be-1795-4728-b8b6-075038cff8ff\") " pod="openshift-ingress/payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9"
Oct 29 22:04:50.299032 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:50.298487    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30ae4be-1795-4728-b8b6-075038cff8ff-plugins-config-volume\") pod \"payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9\" (UID: \"f30ae4be-1795-4728-b8b6-075038cff8ff\") " pod="openshift-ingress/payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9"
Oct 29 22:04:50.318494 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:50.318439    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmgmq\" (UniqueName: \"kubernetes.io/projected/f30ae4be-1795-4728-b8b6-075038cff8ff-kube-api-access-rmgmq\") pod \"payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9\" (UID: \"f30ae4be-1795-4728-b8b6-075038cff8ff\") " pod="openshift-ingress/payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9"
Oct 29 22:04:50.437106 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:50.437074    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9"
Oct 29 22:04:50.544633 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:50.544588    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4"]
Oct 29 22:04:50.549551 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:04:50.549518    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37af084a_d949_4a30_a867_54e3cc3943b9.slice/crio-412a87c279d638f5f6105ec2f521f1f5af809d13458d357db4c32e509404fb00 WatchSource:0}: Error finding container 412a87c279d638f5f6105ec2f521f1f5af809d13458d357db4c32e509404fb00: Status 404 returned error can't find the container with id 412a87c279d638f5f6105ec2f521f1f5af809d13458d357db4c32e509404fb00
Oct 29 22:04:50.750624 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:50.750595    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9"]
Oct 29 22:04:50.751069 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:04:50.750621    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf30ae4be_1795_4728_b8b6_075038cff8ff.slice/crio-acbd2e379e332db25e4d74ac913377cc4d4da2974112059726e4ec2a3804a0e0 WatchSource:0}: Error finding container acbd2e379e332db25e4d74ac913377cc4d4da2974112059726e4ec2a3804a0e0: Status 404 returned error can't find the container with id acbd2e379e332db25e4d74ac913377cc4d4da2974112059726e4ec2a3804a0e0
Oct 29 22:04:51.248460 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:51.248425    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9" event={"ID":"f30ae4be-1795-4728-b8b6-075038cff8ff","Type":"ContainerStarted","Data":"182616628e67876ba3eaf7179da98bf2461cd41341516f1c3e09b188dac4c589"}
Oct 29 22:04:51.248665 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:51.248477    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9" event={"ID":"f30ae4be-1795-4728-b8b6-075038cff8ff","Type":"ContainerStarted","Data":"acbd2e379e332db25e4d74ac913377cc4d4da2974112059726e4ec2a3804a0e0"}
Oct 29 22:04:51.248665 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:51.248644    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9"
Oct 29 22:04:51.250336 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:51.250281    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4" event={"ID":"37af084a-d949-4a30-a867-54e3cc3943b9","Type":"ContainerStarted","Data":"2fc7c496b4ddcb57dba46a3ec0cdbd3ce4182eb095ac0fad37b3ca47e4fce76d"}
Oct 29 22:04:51.250435 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:51.250359    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4"
Oct 29 22:04:51.250435 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:51.250376    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4" event={"ID":"37af084a-d949-4a30-a867-54e3cc3943b9","Type":"ContainerStarted","Data":"412a87c279d638f5f6105ec2f521f1f5af809d13458d357db4c32e509404fb00"}
Oct 29 22:04:51.294034 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:51.293994    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9" podStartSLOduration=2.293980094 podStartE2EDuration="2.293980094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:51.289189915 +0000 UTC m=+1112.362146880" watchObservedRunningTime="2026-10-29 22:04:51.293980094 +0000 UTC m=+1112.366937058"
Oct 29 22:04:51.333418 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:51.333375    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4" podStartSLOduration=2.333362778 podStartE2EDuration="2.333362778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:04:51.33005475 +0000 UTC m=+1112.403011741" watchObservedRunningTime="2026-10-29 22:04:51.333362778 +0000 UTC m=+1112.406319741"
Oct 29 22:04:56.184309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:56.183481    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw"
Oct 29 22:04:56.189883 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:04:56.185491    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c"
Oct 29 22:05:02.320578 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:02.320157    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4"
Oct 29 22:05:02.332125 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:02.331467    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9"
Oct 29 22:05:03.927643 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:03.927604    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c"]
Oct 29 22:05:03.933679 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:03.933657    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c"
Oct 29 22:05:03.942229 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:03.942197    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-openshift-service-ca\") pod \"maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c\" (UID: \"a9d95970-4217-4ac1-ac2e-e05d38b10ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c"
Oct 29 22:05:03.942365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:03.942341    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6z6n\" (UniqueName: \"kubernetes.io/projected/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-kube-api-access-f6z6n\") pod \"maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c\" (UID: \"a9d95970-4217-4ac1-ac2e-e05d38b10ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c"
Oct 29 22:05:03.942645 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:03.942483    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-maas-api-tls\") pod \"maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c\" (UID: \"a9d95970-4217-4ac1-ac2e-e05d38b10ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c"
Oct 29 22:05:03.943003 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:03.942642    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-metrics-certs\") pod \"maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c\" (UID: \"a9d95970-4217-4ac1-ac2e-e05d38b10ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c"
Oct 29 22:05:03.952682 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:03.952655    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f6171d-serving-cert\""
Oct 29 22:05:03.959328 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:03.959272    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c"]
Oct 29 22:05:04.044223 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:04.044192    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-openshift-service-ca\") pod \"maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c\" (UID: \"a9d95970-4217-4ac1-ac2e-e05d38b10ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c"
Oct 29 22:05:04.044466 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:04.044263    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6z6n\" (UniqueName: \"kubernetes.io/projected/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-kube-api-access-f6z6n\") pod \"maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c\" (UID: \"a9d95970-4217-4ac1-ac2e-e05d38b10ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c"
Oct 29 22:05:04.044466 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:04.044387    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-maas-api-tls\") pod \"maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c\" (UID: \"a9d95970-4217-4ac1-ac2e-e05d38b10ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c"
Oct 29 22:05:04.044648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:04.044534    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-metrics-certs\") pod \"maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c\" (UID: \"a9d95970-4217-4ac1-ac2e-e05d38b10ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c"
Oct 29 22:05:04.044970 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:04.044947    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-openshift-service-ca\") pod \"maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c\" (UID: \"a9d95970-4217-4ac1-ac2e-e05d38b10ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c"
Oct 29 22:05:04.046925 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:04.046898    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-metrics-certs\") pod \"maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c\" (UID: \"a9d95970-4217-4ac1-ac2e-e05d38b10ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c"
Oct 29 22:05:04.047053 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:04.047034    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-maas-api-tls\") pod \"maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c\" (UID: \"a9d95970-4217-4ac1-ac2e-e05d38b10ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c"
Oct 29 22:05:04.134654 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:04.134367    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6z6n\" (UniqueName: \"kubernetes.io/projected/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-kube-api-access-f6z6n\") pod \"maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c\" (UID: \"a9d95970-4217-4ac1-ac2e-e05d38b10ea7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c"
Oct 29 22:05:04.255364 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:04.252994    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c"
Oct 29 22:05:04.619736 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:04.619707    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c"]
Oct 29 22:05:04.621057 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:05:04.621032    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d95970_4217_4ac1_ac2e_e05d38b10ea7.slice/crio-19d6df5dba058a3fdd70644baf2ebd5499310bae1cbfc32ca9a3771d0e919a7d WatchSource:0}: Error finding container 19d6df5dba058a3fdd70644baf2ebd5499310bae1cbfc32ca9a3771d0e919a7d: Status 404 returned error can't find the container with id 19d6df5dba058a3fdd70644baf2ebd5499310bae1cbfc32ca9a3771d0e919a7d
Oct 29 22:05:04.622643 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:04.622626    4962 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:05:05.455163 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:05.455125    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c" event={"ID":"a9d95970-4217-4ac1-ac2e-e05d38b10ea7","Type":"ContainerStarted","Data":"f084309b26d4b3411744737e42b935d3609a28f9c097711a0ed6dfe89f651ad0"}
Oct 29 22:05:05.455645 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:05.455196    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c" event={"ID":"a9d95970-4217-4ac1-ac2e-e05d38b10ea7","Type":"ContainerStarted","Data":"19d6df5dba058a3fdd70644baf2ebd5499310bae1cbfc32ca9a3771d0e919a7d"}
Oct 29 22:05:05.455645 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:05.455228    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c"
Oct 29 22:05:05.503831 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:05.503783    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c" podStartSLOduration=2.09177413 podStartE2EDuration="2.503767951s" totalImagesPullingTime="411.993821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:05.498957465 +0000 UTC m=+1126.571914431" watchObservedRunningTime="2026-10-29 22:05:05.503767951 +0000 UTC m=+1126.576724960"
Oct 29 22:05:09.700866 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:09.700835    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"]
Oct 29 22:05:09.706720 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:09.706696    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"
Oct 29 22:05:09.727179 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:05:09.727150    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:05:09.755755 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:09.755717    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"]
Oct 29 22:05:09.801151 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:09.801123    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-380eea-serving-cert\""
Oct 29 22:05:09.809693 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:09.809663    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-metrics-certs\") pod \"maas-api-e2e-api-a-380eea-5f4984df5-cn25x\" (UID: \"fb5e9803-65bc-4b86-815b-3f1bc323b3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"
Oct 29 22:05:09.809822 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:09.809780    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-openshift-service-ca\") pod \"maas-api-e2e-api-a-380eea-5f4984df5-cn25x\" (UID: \"fb5e9803-65bc-4b86-815b-3f1bc323b3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"
Oct 29 22:05:09.810024 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:09.810001    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-maas-api-tls\") pod \"maas-api-e2e-api-a-380eea-5f4984df5-cn25x\" (UID: \"fb5e9803-65bc-4b86-815b-3f1bc323b3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"
Oct 29 22:05:09.810240 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:09.810220    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stbkx\" (UniqueName: \"kubernetes.io/projected/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-kube-api-access-stbkx\") pod \"maas-api-e2e-api-a-380eea-5f4984df5-cn25x\" (UID: \"fb5e9803-65bc-4b86-815b-3f1bc323b3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"
Oct 29 22:05:09.911778 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:09.911749    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-maas-api-tls\") pod \"maas-api-e2e-api-a-380eea-5f4984df5-cn25x\" (UID: \"fb5e9803-65bc-4b86-815b-3f1bc323b3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"
Oct 29 22:05:09.911940 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:09.911911    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stbkx\" (UniqueName: \"kubernetes.io/projected/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-kube-api-access-stbkx\") pod \"maas-api-e2e-api-a-380eea-5f4984df5-cn25x\" (UID: \"fb5e9803-65bc-4b86-815b-3f1bc323b3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"
Oct 29 22:05:09.912005 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:09.911990    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-metrics-certs\") pod \"maas-api-e2e-api-a-380eea-5f4984df5-cn25x\" (UID: \"fb5e9803-65bc-4b86-815b-3f1bc323b3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"
Oct 29 22:05:09.912064 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:05:09.911915    4962 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-serving-cert: secret "maas-api-e2e-api-a-380eea-serving-cert" not found
Oct 29 22:05:09.912114 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:09.912075    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-openshift-service-ca\") pod \"maas-api-e2e-api-a-380eea-5f4984df5-cn25x\" (UID: \"fb5e9803-65bc-4b86-815b-3f1bc323b3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"
Oct 29 22:05:09.912170 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:05:09.912116    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-maas-api-tls podName:fb5e9803-65bc-4b86-815b-3f1bc323b3bb nodeName:}" failed. No retries permitted until 2026-10-29 22:05:10.412094095 +0000 UTC m=+1131.485051054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-maas-api-tls") pod "maas-api-e2e-api-a-380eea-5f4984df5-cn25x" (UID: "fb5e9803-65bc-4b86-815b-3f1bc323b3bb") : secret "maas-api-e2e-api-a-380eea-serving-cert" not found
Oct 29 22:05:09.912752 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:09.912731    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-openshift-service-ca\") pod \"maas-api-e2e-api-a-380eea-5f4984df5-cn25x\" (UID: \"fb5e9803-65bc-4b86-815b-3f1bc323b3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"
Oct 29 22:05:09.915163 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:09.915142    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-metrics-certs\") pod \"maas-api-e2e-api-a-380eea-5f4984df5-cn25x\" (UID: \"fb5e9803-65bc-4b86-815b-3f1bc323b3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"
Oct 29 22:05:09.973370 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:09.967691    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stbkx\" (UniqueName: \"kubernetes.io/projected/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-kube-api-access-stbkx\") pod \"maas-api-e2e-api-a-380eea-5f4984df5-cn25x\" (UID: \"fb5e9803-65bc-4b86-815b-3f1bc323b3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"
Oct 29 22:05:10.422968 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:10.422937    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-maas-api-tls\") pod \"maas-api-e2e-api-a-380eea-5f4984df5-cn25x\" (UID: \"fb5e9803-65bc-4b86-815b-3f1bc323b3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"
Oct 29 22:05:10.425429 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:10.425404    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-maas-api-tls\") pod \"maas-api-e2e-api-a-380eea-5f4984df5-cn25x\" (UID: \"fb5e9803-65bc-4b86-815b-3f1bc323b3bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"
Oct 29 22:05:10.624451 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:10.624421    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"
Oct 29 22:05:10.970129 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:10.969931    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"]
Oct 29 22:05:10.970661 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:05:10.970631    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5e9803_65bc_4b86_815b_3f1bc323b3bb.slice/crio-81f88d01d297802c33d499734bd558e11431f6783cda253349b0a45eb8861be5 WatchSource:0}: Error finding container 81f88d01d297802c33d499734bd558e11431f6783cda253349b0a45eb8861be5: Status 404 returned error can't find the container with id 81f88d01d297802c33d499734bd558e11431f6783cda253349b0a45eb8861be5
Oct 29 22:05:11.471646 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:11.471622    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c"
Oct 29 22:05:11.582654 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:11.582625    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x" event={"ID":"fb5e9803-65bc-4b86-815b-3f1bc323b3bb","Type":"ContainerStarted","Data":"81f88d01d297802c33d499734bd558e11431f6783cda253349b0a45eb8861be5"}
Oct 29 22:05:12.594150 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:12.594114    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x" event={"ID":"fb5e9803-65bc-4b86-815b-3f1bc323b3bb","Type":"ContainerStarted","Data":"4b55130cebce2b54497dd37035b435da89cc38491b19adb1802ce1d15fbf644e"}
Oct 29 22:05:12.594584 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:12.594350    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"
Oct 29 22:05:12.637651 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:12.637605    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x" podStartSLOduration=3.121653573 podStartE2EDuration="3.637591049s" totalImagesPullingTime="515.937476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:12.637004971 +0000 UTC m=+1133.709961937" watchObservedRunningTime="2026-10-29 22:05:12.637591049 +0000 UTC m=+1133.710548020"
Oct 29 22:05:16.520233 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:16.520144    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx"]
Oct 29 22:05:18.607393 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:18.607360    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"
Oct 29 22:05:21.550393 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:21.546637    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx" podUID="d0173eb5-ad53-4ba5-8abd-6bfe9d27d989" containerName="payload-pre-processing" containerID="cri-o://ef00db8e5b187649d7b9cf8d32e7d4fc37df16774a77fa71d5482023de3de251" gracePeriod=25
Oct 29 22:05:22.545351 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:22.545327    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx"
Oct 29 22:05:22.652135 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:22.652101    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0173eb5-ad53-4ba5-8abd-6bfe9d27d989-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0173eb5-ad53-4ba5-8abd-6bfe9d27d989-plugins-config-volume\") pod \"d0173eb5-ad53-4ba5-8abd-6bfe9d27d989\" (UID: \"d0173eb5-ad53-4ba5-8abd-6bfe9d27d989\") "
Oct 29 22:05:22.652567 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:22.652267    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0173eb5-ad53-4ba5-8abd-6bfe9d27d989-kube-api-access-c9tcr\" (UniqueName: \"kubernetes.io/projected/d0173eb5-ad53-4ba5-8abd-6bfe9d27d989-kube-api-access-c9tcr\") pod \"d0173eb5-ad53-4ba5-8abd-6bfe9d27d989\" (UID: \"d0173eb5-ad53-4ba5-8abd-6bfe9d27d989\") "
Oct 29 22:05:22.652626 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:22.652559    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0173eb5-ad53-4ba5-8abd-6bfe9d27d989-plugins-config-volume" pod "d0173eb5-ad53-4ba5-8abd-6bfe9d27d989" (UID: "d0173eb5-ad53-4ba5-8abd-6bfe9d27d989"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:05:22.653135 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:22.653110    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0173eb5-ad53-4ba5-8abd-6bfe9d27d989-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:05:22.654805 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:22.654766    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0173eb5-ad53-4ba5-8abd-6bfe9d27d989-kube-api-access-c9tcr" pod "d0173eb5-ad53-4ba5-8abd-6bfe9d27d989" (UID: "d0173eb5-ad53-4ba5-8abd-6bfe9d27d989"). InnerVolumeSpecName "kube-api-access-c9tcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:05:22.754686 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:22.754652    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9tcr\" (UniqueName: \"kubernetes.io/projected/d0173eb5-ad53-4ba5-8abd-6bfe9d27d989-kube-api-access-c9tcr\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:05:22.904358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:22.904317    4962 generic.go:356] "Generic (PLEG): container finished" podID="d0173eb5-ad53-4ba5-8abd-6bfe9d27d989" containerID="ef00db8e5b187649d7b9cf8d32e7d4fc37df16774a77fa71d5482023de3de251" exitCode=0
Oct 29 22:05:22.904527 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:22.904363    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx"
Oct 29 22:05:22.904527 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:22.904370    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx" event={"ID":"d0173eb5-ad53-4ba5-8abd-6bfe9d27d989","Type":"ContainerDied","Data":"ef00db8e5b187649d7b9cf8d32e7d4fc37df16774a77fa71d5482023de3de251"}
Oct 29 22:05:22.904527 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:22.904449    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx" event={"ID":"d0173eb5-ad53-4ba5-8abd-6bfe9d27d989","Type":"ContainerDied","Data":"27856133ff556a2e71c5c53a1650db53b0d78e5d04e4f88bb9824d304431e40f"}
Oct 29 22:05:22.904527 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:22.904471    4962 scope.go:122] "RemoveContainer" containerID="ef00db8e5b187649d7b9cf8d32e7d4fc37df16774a77fa71d5482023de3de251"
Oct 29 22:05:22.916555 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:22.916535    4962 scope.go:122] "RemoveContainer" containerID="ef00db8e5b187649d7b9cf8d32e7d4fc37df16774a77fa71d5482023de3de251"
Oct 29 22:05:22.916803 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:05:22.916782    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef00db8e5b187649d7b9cf8d32e7d4fc37df16774a77fa71d5482023de3de251\": container with ID starting with ef00db8e5b187649d7b9cf8d32e7d4fc37df16774a77fa71d5482023de3de251 not found: ID does not exist" containerID="ef00db8e5b187649d7b9cf8d32e7d4fc37df16774a77fa71d5482023de3de251"
Oct 29 22:05:22.916849 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:22.916811    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef00db8e5b187649d7b9cf8d32e7d4fc37df16774a77fa71d5482023de3de251"} err="failed to get container status \"ef00db8e5b187649d7b9cf8d32e7d4fc37df16774a77fa71d5482023de3de251\": rpc error: code = NotFound desc = could not find container \"ef00db8e5b187649d7b9cf8d32e7d4fc37df16774a77fa71d5482023de3de251\": container with ID starting with ef00db8e5b187649d7b9cf8d32e7d4fc37df16774a77fa71d5482023de3de251 not found: ID does not exist"
Oct 29 22:05:22.960496 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:22.960427    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx"]
Oct 29 22:05:22.971155 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:22.971135    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6df539-5576568c-w67tx"]
Oct 29 22:05:23.059643 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:23.059613    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0173eb5-ad53-4ba5-8abd-6bfe9d27d989" path="/var/lib/kubelet/pods/d0173eb5-ad53-4ba5-8abd-6bfe9d27d989/volumes"
Oct 29 22:05:29.003076 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:29.003045    4962 generic.go:356] "Generic (PLEG): container finished" podID="630849f8-61af-40a9-80b9-7f4ae5a4ee6e" containerID="52700b1df6d64125547285c51c1e9026f981585a8ffa49ac9fc2ee9e31661c7e" exitCode=0
Oct 29 22:05:29.003534 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:29.003119    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k" event={"ID":"630849f8-61af-40a9-80b9-7f4ae5a4ee6e","Type":"ContainerDied","Data":"52700b1df6d64125547285c51c1e9026f981585a8ffa49ac9fc2ee9e31661c7e"}
Oct 29 22:05:29.003664 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:29.003645    4962 scope.go:122] "RemoveContainer" containerID="52700b1df6d64125547285c51c1e9026f981585a8ffa49ac9fc2ee9e31661c7e"
Oct 29 22:05:29.005503 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:29.005476    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:05:29.005564 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:29.005548    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:05:29.005603 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:29.005581    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:05:30.021924 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:30.021886    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k" event={"ID":"630849f8-61af-40a9-80b9-7f4ae5a4ee6e","Type":"ContainerStarted","Data":"df85268bcd864b52d4dc77ca2e2d74d4a2274cda41990c4a866581e17b67ab43"}
Oct 29 22:05:30.541125 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:30.541088    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:05:30.542543 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:30.542514    4962 patch_prober.go:29] interesting pod/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 29 22:05:30.542675 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:30.542566    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k" podUID="630849f8-61af-40a9-80b9-7f4ae5a4ee6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 29 22:05:31.549931 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:31.549890    4962 patch_prober.go:29] interesting pod/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 29 22:05:31.551857 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:31.551812    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k" podUID="630849f8-61af-40a9-80b9-7f4ae5a4ee6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 29 22:05:32.301196 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:32.301160    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:05:32.541914 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:32.541877    4962 patch_prober.go:29] interesting pod/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 29 22:05:32.542078 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:32.541949    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k" podUID="630849f8-61af-40a9-80b9-7f4ae5a4ee6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 29 22:05:33.553848 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:33.553815    4962 patch_prober.go:29] interesting pod/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 29 22:05:33.554495 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:33.554461    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k" podUID="630849f8-61af-40a9-80b9-7f4ae5a4ee6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 29 22:05:34.541587 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:34.541555    4962 patch_prober.go:29] interesting pod/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 29 22:05:34.541749 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:34.541608    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k" podUID="630849f8-61af-40a9-80b9-7f4ae5a4ee6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 29 22:05:35.552117 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:35.551090    4962 patch_prober.go:29] interesting pod/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body=
Oct 29 22:05:35.552117 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:35.551156    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k" podUID="630849f8-61af-40a9-80b9-7f4ae5a4ee6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused"
Oct 29 22:05:36.648475 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:36.648438    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:05:36.650134 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:36.650107    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:05:37.222204 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:37.222166    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk"]
Oct 29 22:05:37.223100 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:37.223080    4962 cpu_manager.go:414] "containerMap: removing container" podUID="d0173eb5-ad53-4ba5-8abd-6bfe9d27d989" containerName="payload-pre-processing"
Oct 29 22:05:37.223171 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:37.223101    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0173eb5-ad53-4ba5-8abd-6bfe9d27d989" containerName="payload-pre-processing"
Oct 29 22:05:37.223244 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:37.223234    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0173eb5-ad53-4ba5-8abd-6bfe9d27d989" containerName="payload-pre-processing"
Oct 29 22:05:37.227480 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:37.227460    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk"
Oct 29 22:05:37.250163 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:37.250137    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-dd38c2-serving-cert\""
Oct 29 22:05:37.257664 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:37.257636    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk"]
Oct 29 22:05:37.330522 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:37.330485    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqs6t\" (UniqueName: \"kubernetes.io/projected/1da3b722-7a00-41ea-969d-d1cfe43314a9-kube-api-access-nqs6t\") pod \"maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk\" (UID: \"1da3b722-7a00-41ea-969d-d1cfe43314a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk"
Oct 29 22:05:37.330696 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:37.330677    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da3b722-7a00-41ea-969d-d1cfe43314a9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk\" (UID: \"1da3b722-7a00-41ea-969d-d1cfe43314a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk"
Oct 29 22:05:37.330836 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:37.330816    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da3b722-7a00-41ea-969d-d1cfe43314a9-metrics-certs\") pod \"maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk\" (UID: \"1da3b722-7a00-41ea-969d-d1cfe43314a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk"
Oct 29 22:05:37.330895 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:37.330881    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1da3b722-7a00-41ea-969d-d1cfe43314a9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk\" (UID: \"1da3b722-7a00-41ea-969d-d1cfe43314a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk"
Oct 29 22:05:37.431961 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:37.431925    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1da3b722-7a00-41ea-969d-d1cfe43314a9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk\" (UID: \"1da3b722-7a00-41ea-969d-d1cfe43314a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk"
Oct 29 22:05:37.432144 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:37.431978    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqs6t\" (UniqueName: \"kubernetes.io/projected/1da3b722-7a00-41ea-969d-d1cfe43314a9-kube-api-access-nqs6t\") pod \"maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk\" (UID: \"1da3b722-7a00-41ea-969d-d1cfe43314a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk"
Oct 29 22:05:37.432209 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:37.432181    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da3b722-7a00-41ea-969d-d1cfe43314a9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk\" (UID: \"1da3b722-7a00-41ea-969d-d1cfe43314a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk"
Oct 29 22:05:37.432380 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:37.432363    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da3b722-7a00-41ea-969d-d1cfe43314a9-metrics-certs\") pod \"maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk\" (UID: \"1da3b722-7a00-41ea-969d-d1cfe43314a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk"
Oct 29 22:05:37.432975 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:37.432955    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da3b722-7a00-41ea-969d-d1cfe43314a9-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk\" (UID: \"1da3b722-7a00-41ea-969d-d1cfe43314a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk"
Oct 29 22:05:37.434391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:37.434364    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1da3b722-7a00-41ea-969d-d1cfe43314a9-maas-api-tls\") pod \"maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk\" (UID: \"1da3b722-7a00-41ea-969d-d1cfe43314a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk"
Oct 29 22:05:37.434666 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:37.434647    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da3b722-7a00-41ea-969d-d1cfe43314a9-metrics-certs\") pod \"maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk\" (UID: \"1da3b722-7a00-41ea-969d-d1cfe43314a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk"
Oct 29 22:05:37.484936 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:37.484150    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqs6t\" (UniqueName: \"kubernetes.io/projected/1da3b722-7a00-41ea-969d-d1cfe43314a9-kube-api-access-nqs6t\") pod \"maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk\" (UID: \"1da3b722-7a00-41ea-969d-d1cfe43314a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk"
Oct 29 22:05:37.546986 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:37.546592    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk"
Oct 29 22:05:38.220383 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:38.218520    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk"]
Oct 29 22:05:39.320672 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:39.320631    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk" event={"ID":"1da3b722-7a00-41ea-969d-d1cfe43314a9","Type":"ContainerStarted","Data":"514236a2995511da9e85d53f96a84f2ac36b305130452b862a2a752e8f84ad97"}
Oct 29 22:05:39.322798 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:39.322772    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk"
Oct 29 22:05:39.323068 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:39.323047    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk" event={"ID":"1da3b722-7a00-41ea-969d-d1cfe43314a9","Type":"ContainerStarted","Data":"2176cb11885640f8e60aefbf0f21c9639544b65fb83505c5dad0becfecba7bfa"}
Oct 29 22:05:39.435606 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:39.435469    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk" podStartSLOduration=1.994984576 podStartE2EDuration="2.435447654s" totalImagesPullingTime="440.463078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:39.429094283 +0000 UTC m=+1160.502051467" watchObservedRunningTime="2026-10-29 22:05:39.435447654 +0000 UTC m=+1160.508404619"
Oct 29 22:05:39.976799 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:05:39.954414    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:05:46.429173 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:46.429143    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk"
Oct 29 22:05:46.836530 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:46.836456    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj"]
Oct 29 22:05:46.842617 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:46.842592    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj"
Oct 29 22:05:46.861506 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:46.861484    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2d53cd\""
Oct 29 22:05:46.868171 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:46.868146    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj"]
Oct 29 22:05:47.035384 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:47.035342    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb8m2\" (UniqueName: \"kubernetes.io/projected/45b4cce5-7740-4811-9191-9bfddb5e3fac-kube-api-access-rb8m2\") pod \"payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj\" (UID: \"45b4cce5-7740-4811-9191-9bfddb5e3fac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj"
Oct 29 22:05:47.036095 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:47.036065    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b4cce5-7740-4811-9191-9bfddb5e3fac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj\" (UID: \"45b4cce5-7740-4811-9191-9bfddb5e3fac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj"
Oct 29 22:05:47.160115 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:47.160079    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb8m2\" (UniqueName: \"kubernetes.io/projected/45b4cce5-7740-4811-9191-9bfddb5e3fac-kube-api-access-rb8m2\") pod \"payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj\" (UID: \"45b4cce5-7740-4811-9191-9bfddb5e3fac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj"
Oct 29 22:05:47.177225 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:47.177190    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b4cce5-7740-4811-9191-9bfddb5e3fac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj\" (UID: \"45b4cce5-7740-4811-9191-9bfddb5e3fac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj"
Oct 29 22:05:47.179393 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:47.178575    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b4cce5-7740-4811-9191-9bfddb5e3fac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj\" (UID: \"45b4cce5-7740-4811-9191-9bfddb5e3fac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj"
Oct 29 22:05:47.252079 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:47.252010    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb8m2\" (UniqueName: \"kubernetes.io/projected/45b4cce5-7740-4811-9191-9bfddb5e3fac-kube-api-access-rb8m2\") pod \"payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj\" (UID: \"45b4cce5-7740-4811-9191-9bfddb5e3fac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj"
Oct 29 22:05:47.458279 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:47.458197    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj"
Oct 29 22:05:47.788446 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:05:47.788415    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b4cce5_7740_4811_9191_9bfddb5e3fac.slice/crio-65b1e218856dc9f6c94f97714d3d504245f927341e99ff19d0688da84e6b3227 WatchSource:0}: Error finding container 65b1e218856dc9f6c94f97714d3d504245f927341e99ff19d0688da84e6b3227: Status 404 returned error can't find the container with id 65b1e218856dc9f6c94f97714d3d504245f927341e99ff19d0688da84e6b3227
Oct 29 22:05:47.792320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:47.792269    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj"]
Oct 29 22:05:48.545323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:48.544190    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj" event={"ID":"45b4cce5-7740-4811-9191-9bfddb5e3fac","Type":"ContainerStarted","Data":"9ed2aac9a09967bf8a8e9d07727c9efb6d18b181612d36ad7f0560308fcde426"}
Oct 29 22:05:48.545323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:48.544262    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj" event={"ID":"45b4cce5-7740-4811-9191-9bfddb5e3fac","Type":"ContainerStarted","Data":"65b1e218856dc9f6c94f97714d3d504245f927341e99ff19d0688da84e6b3227"}
Oct 29 22:05:48.545917 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:48.545654    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj"
Oct 29 22:05:48.596744 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:48.595960    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj" podStartSLOduration=2.595940303 podStartE2EDuration="2.595940303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:05:48.594593323 +0000 UTC m=+1169.667550286" watchObservedRunningTime="2026-10-29 22:05:48.595940303 +0000 UTC m=+1169.668897271"
Oct 29 22:05:56.762092 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:56.762057    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"]
Oct 29 22:05:56.762664 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:56.762335    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk" podUID="d5e9b803-7f4a-42eb-9535-9cd54d1ae76a" containerName="maas-api" containerID="cri-o://ef7ea00647a275fbbd8e09fd87523f42581ef3e803a7f90b8cfb2740a74ce381" gracePeriod=30
Oct 29 22:05:57.378343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:57.378253    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"
Oct 29 22:05:57.416489 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:57.412891    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-openshift-service-ca\") pod \"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a\" (UID: \"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a\") "
Oct 29 22:05:57.416489 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:57.412966    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-maas-api-tls\") pod \"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a\" (UID: \"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a\") "
Oct 29 22:05:57.416489 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:57.413278    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-metrics-certs\") pod \"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a\" (UID: \"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a\") "
Oct 29 22:05:57.416489 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:57.413515    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-kube-api-access-lzcm7\" (UniqueName: \"kubernetes.io/projected/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-kube-api-access-lzcm7\") pod \"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a\" (UID: \"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a\") "
Oct 29 22:05:57.425634 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:57.424958    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-openshift-service-ca" pod "d5e9b803-7f4a-42eb-9535-9cd54d1ae76a" (UID: "d5e9b803-7f4a-42eb-9535-9cd54d1ae76a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:05:57.428913 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:57.428882    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-metrics-certs" pod "d5e9b803-7f4a-42eb-9535-9cd54d1ae76a" (UID: "d5e9b803-7f4a-42eb-9535-9cd54d1ae76a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:05:57.431515 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:57.431471    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-maas-api-tls" pod "d5e9b803-7f4a-42eb-9535-9cd54d1ae76a" (UID: "d5e9b803-7f4a-42eb-9535-9cd54d1ae76a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:05:57.452253 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:57.452094    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-kube-api-access-lzcm7" pod "d5e9b803-7f4a-42eb-9535-9cd54d1ae76a" (UID: "d5e9b803-7f4a-42eb-9535-9cd54d1ae76a"). InnerVolumeSpecName "kube-api-access-lzcm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:05:57.791698 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:57.791595    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzcm7\" (UniqueName: \"kubernetes.io/projected/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-kube-api-access-lzcm7\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:05:57.792302 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:57.792261    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:05:57.792423 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:57.792409    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:05:57.792523 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:57.792511    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:05:58.012669 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:58.012627    4962 generic.go:356] "Generic (PLEG): container finished" podID="d5e9b803-7f4a-42eb-9535-9cd54d1ae76a" containerID="ef7ea00647a275fbbd8e09fd87523f42581ef3e803a7f90b8cfb2740a74ce381" exitCode=0
Oct 29 22:05:58.012872 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:58.012694    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk" event={"ID":"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a","Type":"ContainerDied","Data":"ef7ea00647a275fbbd8e09fd87523f42581ef3e803a7f90b8cfb2740a74ce381"}
Oct 29 22:05:58.012872 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:58.012803    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk" event={"ID":"d5e9b803-7f4a-42eb-9535-9cd54d1ae76a","Type":"ContainerDied","Data":"a67f282611fe889b7ddf6fa111f9c9a78132cd9e025db83188d11533fa6f45d3"}
Oct 29 22:05:58.012872 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:58.012836    4962 scope.go:122] "RemoveContainer" containerID="ef7ea00647a275fbbd8e09fd87523f42581ef3e803a7f90b8cfb2740a74ce381"
Oct 29 22:05:58.013053 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:58.013006    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"
Oct 29 22:05:58.024842 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:58.024820    4962 scope.go:122] "RemoveContainer" containerID="ef7ea00647a275fbbd8e09fd87523f42581ef3e803a7f90b8cfb2740a74ce381"
Oct 29 22:05:58.025141 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:05:58.025121    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef7ea00647a275fbbd8e09fd87523f42581ef3e803a7f90b8cfb2740a74ce381\": container with ID starting with ef7ea00647a275fbbd8e09fd87523f42581ef3e803a7f90b8cfb2740a74ce381 not found: ID does not exist" containerID="ef7ea00647a275fbbd8e09fd87523f42581ef3e803a7f90b8cfb2740a74ce381"
Oct 29 22:05:58.025217 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:58.025153    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef7ea00647a275fbbd8e09fd87523f42581ef3e803a7f90b8cfb2740a74ce381"} err="failed to get container status \"ef7ea00647a275fbbd8e09fd87523f42581ef3e803a7f90b8cfb2740a74ce381\": rpc error: code = NotFound desc = could not find container \"ef7ea00647a275fbbd8e09fd87523f42581ef3e803a7f90b8cfb2740a74ce381\": container with ID starting with ef7ea00647a275fbbd8e09fd87523f42581ef3e803a7f90b8cfb2740a74ce381 not found: ID does not exist"
Oct 29 22:05:58.068357 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:58.068255    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"]
Oct 29 22:05:58.079544 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:58.079519    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6df539-7787bbf957-5gjfk"]
Oct 29 22:05:59.062173 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:05:59.062136    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5e9b803-7f4a-42eb-9535-9cd54d1ae76a" path="/var/lib/kubelet/pods/d5e9b803-7f4a-42eb-9535-9cd54d1ae76a/volumes"
Oct 29 22:06:00.751179 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:00.751152    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj"
Oct 29 22:06:02.973643 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:02.973610    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp"]
Oct 29 22:06:02.974573 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:02.974551    4962 cpu_manager.go:414] "containerMap: removing container" podUID="d5e9b803-7f4a-42eb-9535-9cd54d1ae76a" containerName="maas-api"
Oct 29 22:06:02.974573 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:02.974572    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e9b803-7f4a-42eb-9535-9cd54d1ae76a" containerName="maas-api"
Oct 29 22:06:02.974733 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:02.974721    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e9b803-7f4a-42eb-9535-9cd54d1ae76a" containerName="maas-api"
Oct 29 22:06:02.980667 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:02.980644    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp"
Oct 29 22:06:03.000281 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:03.000255    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a26f29-serving-cert\""
Oct 29 22:06:03.005105 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:03.005076    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp"]
Oct 29 22:06:03.111629 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:03.111601    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e00f0b9-3982-424a-8c78-7037bcda1f79-metrics-certs\") pod \"maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp\" (UID: \"6e00f0b9-3982-424a-8c78-7037bcda1f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp"
Oct 29 22:06:03.111881 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:03.111858    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e00f0b9-3982-424a-8c78-7037bcda1f79-maas-api-tls\") pod \"maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp\" (UID: \"6e00f0b9-3982-424a-8c78-7037bcda1f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp"
Oct 29 22:06:03.112059 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:03.112037    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e00f0b9-3982-424a-8c78-7037bcda1f79-openshift-service-ca\") pod \"maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp\" (UID: \"6e00f0b9-3982-424a-8c78-7037bcda1f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp"
Oct 29 22:06:03.112136 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:03.112114    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hddt6\" (UniqueName: \"kubernetes.io/projected/6e00f0b9-3982-424a-8c78-7037bcda1f79-kube-api-access-hddt6\") pod \"maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp\" (UID: \"6e00f0b9-3982-424a-8c78-7037bcda1f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp"
Oct 29 22:06:03.213994 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:03.213954    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e00f0b9-3982-424a-8c78-7037bcda1f79-maas-api-tls\") pod \"maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp\" (UID: \"6e00f0b9-3982-424a-8c78-7037bcda1f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp"
Oct 29 22:06:03.214191 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:03.214056    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e00f0b9-3982-424a-8c78-7037bcda1f79-openshift-service-ca\") pod \"maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp\" (UID: \"6e00f0b9-3982-424a-8c78-7037bcda1f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp"
Oct 29 22:06:03.214191 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:03.214090    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hddt6\" (UniqueName: \"kubernetes.io/projected/6e00f0b9-3982-424a-8c78-7037bcda1f79-kube-api-access-hddt6\") pod \"maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp\" (UID: \"6e00f0b9-3982-424a-8c78-7037bcda1f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp"
Oct 29 22:06:03.214341 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:03.214241    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e00f0b9-3982-424a-8c78-7037bcda1f79-metrics-certs\") pod \"maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp\" (UID: \"6e00f0b9-3982-424a-8c78-7037bcda1f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp"
Oct 29 22:06:03.214919 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:03.214890    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e00f0b9-3982-424a-8c78-7037bcda1f79-openshift-service-ca\") pod \"maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp\" (UID: \"6e00f0b9-3982-424a-8c78-7037bcda1f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp"
Oct 29 22:06:03.216623 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:03.216593    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e00f0b9-3982-424a-8c78-7037bcda1f79-metrics-certs\") pod \"maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp\" (UID: \"6e00f0b9-3982-424a-8c78-7037bcda1f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp"
Oct 29 22:06:03.216742 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:03.216691    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e00f0b9-3982-424a-8c78-7037bcda1f79-maas-api-tls\") pod \"maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp\" (UID: \"6e00f0b9-3982-424a-8c78-7037bcda1f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp"
Oct 29 22:06:03.238137 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:03.238078    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hddt6\" (UniqueName: \"kubernetes.io/projected/6e00f0b9-3982-424a-8c78-7037bcda1f79-kube-api-access-hddt6\") pod \"maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp\" (UID: \"6e00f0b9-3982-424a-8c78-7037bcda1f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp"
Oct 29 22:06:03.301729 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:03.301689    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp"
Oct 29 22:06:03.822132 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:06:03.822092    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e00f0b9_3982_424a_8c78_7037bcda1f79.slice/crio-e9cd9e4b59c703af805664737fbda7059ecf834d250843802b9c485f5d3bcd15 WatchSource:0}: Error finding container e9cd9e4b59c703af805664737fbda7059ecf834d250843802b9c485f5d3bcd15: Status 404 returned error can't find the container with id e9cd9e4b59c703af805664737fbda7059ecf834d250843802b9c485f5d3bcd15
Oct 29 22:06:03.845110 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:03.844627    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp"]
Oct 29 22:06:04.062764 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:04.062727    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp" event={"ID":"6e00f0b9-3982-424a-8c78-7037bcda1f79","Type":"ContainerStarted","Data":"e9cd9e4b59c703af805664737fbda7059ecf834d250843802b9c485f5d3bcd15"}
Oct 29 22:06:05.073703 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:05.073662    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp" event={"ID":"6e00f0b9-3982-424a-8c78-7037bcda1f79","Type":"ContainerStarted","Data":"d2d6347a492ec3a65b34d4b3cc153cf0a095101eb5ddf24d852f5db521bd90ae"}
Oct 29 22:06:05.074084 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:05.073737    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp"
Oct 29 22:06:05.121151 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:05.121100    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp" podStartSLOduration=2.481779541 podStartE2EDuration="3.121086169s" totalImagesPullingTime="639.306628ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:05.116121667 +0000 UTC m=+1186.189078631" watchObservedRunningTime="2026-10-29 22:06:05.121086169 +0000 UTC m=+1186.194043151"
Oct 29 22:06:06.445310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:06.434973    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"]
Oct 29 22:06:06.445310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:06.435222    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k" podUID="630849f8-61af-40a9-80b9-7f4ae5a4ee6e" containerName="istio-proxy" containerID="cri-o://df85268bcd864b52d4dc77ca2e2d74d4a2274cda41990c4a866581e17b67ab43" gracePeriod=30
Oct 29 22:06:09.725408 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:06:09.725371    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:06:11.199680 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:11.199649    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp"
Oct 29 22:06:11.970888 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:11.970864    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:06:12.054705 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.054625    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-kube-api-access-8msr9\" (UniqueName: \"kubernetes.io/projected/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-kube-api-access-8msr9\") pod \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") "
Oct 29 22:06:12.054873 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.054711    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-credential-socket\") pod \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") "
Oct 29 22:06:12.054873 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.054797    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-podinfo\") pod \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") "
Oct 29 22:06:12.054873 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.054823    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-envoy\") pod \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") "
Oct 29 22:06:12.055061 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.054944    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-token\" (UniqueName: \"kubernetes.io/projected/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-token\") pod \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") "
Oct 29 22:06:12.055121 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.055075    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-workload-socket\") pod \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") "
Oct 29 22:06:12.055178 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.055140    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-data\") pod \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") "
Oct 29 22:06:12.055253 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.055237    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istiod-ca-cert\") pod \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") "
Oct 29 22:06:12.055323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.055275    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-workload-certs\") pod \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\" (UID: \"630849f8-61af-40a9-80b9-7f4ae5a4ee6e\") "
Oct 29 22:06:12.055391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.055138    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-credential-socket" pod "630849f8-61af-40a9-80b9-7f4ae5a4ee6e" (UID: "630849f8-61af-40a9-80b9-7f4ae5a4ee6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:06:12.055455 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.055385    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-data" pod "630849f8-61af-40a9-80b9-7f4ae5a4ee6e" (UID: "630849f8-61af-40a9-80b9-7f4ae5a4ee6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:06:12.055455 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.055364    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-workload-socket" pod "630849f8-61af-40a9-80b9-7f4ae5a4ee6e" (UID: "630849f8-61af-40a9-80b9-7f4ae5a4ee6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:06:12.055710 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.055686    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-workload-certs" pod "630849f8-61af-40a9-80b9-7f4ae5a4ee6e" (UID: "630849f8-61af-40a9-80b9-7f4ae5a4ee6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:06:12.055775 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.055713    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istiod-ca-cert" pod "630849f8-61af-40a9-80b9-7f4ae5a4ee6e" (UID: "630849f8-61af-40a9-80b9-7f4ae5a4ee6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:06:12.056010 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.055986    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:06:12.056074 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.056015    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:06:12.056074 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.056030    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:06:12.056074 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.056043    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:06:12.056074 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.056056    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:06:12.058675 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.058647    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-podinfo" pod "630849f8-61af-40a9-80b9-7f4ae5a4ee6e" (UID: "630849f8-61af-40a9-80b9-7f4ae5a4ee6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:06:12.058866 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.058843    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-token" pod "630849f8-61af-40a9-80b9-7f4ae5a4ee6e" (UID: "630849f8-61af-40a9-80b9-7f4ae5a4ee6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:06:12.060016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.059986    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-envoy" pod "630849f8-61af-40a9-80b9-7f4ae5a4ee6e" (UID: "630849f8-61af-40a9-80b9-7f4ae5a4ee6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:06:12.060231 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.060209    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-kube-api-access-8msr9" pod "630849f8-61af-40a9-80b9-7f4ae5a4ee6e" (UID: "630849f8-61af-40a9-80b9-7f4ae5a4ee6e"). InnerVolumeSpecName "kube-api-access-8msr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:06:12.157448 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.157418    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:06:12.157448 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.157446    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:06:12.157905 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.157459    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:06:12.157905 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.157473    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8msr9\" (UniqueName: \"kubernetes.io/projected/630849f8-61af-40a9-80b9-7f4ae5a4ee6e-kube-api-access-8msr9\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:06:12.214064 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.214033    4962 generic.go:356] "Generic (PLEG): container finished" podID="630849f8-61af-40a9-80b9-7f4ae5a4ee6e" containerID="df85268bcd864b52d4dc77ca2e2d74d4a2274cda41990c4a866581e17b67ab43" exitCode=0
Oct 29 22:06:12.214489 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.214090    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"
Oct 29 22:06:12.214489 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.214108    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k" event={"ID":"630849f8-61af-40a9-80b9-7f4ae5a4ee6e","Type":"ContainerDied","Data":"df85268bcd864b52d4dc77ca2e2d74d4a2274cda41990c4a866581e17b67ab43"}
Oct 29 22:06:12.214489 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.214167    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k" event={"ID":"630849f8-61af-40a9-80b9-7f4ae5a4ee6e","Type":"ContainerDied","Data":"d71f8af3f5bfc7362794682bb6c1cf51ad9db43f599b0bb696d46db30795ee55"}
Oct 29 22:06:12.214489 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.214190    4962 scope.go:122] "RemoveContainer" containerID="df85268bcd864b52d4dc77ca2e2d74d4a2274cda41990c4a866581e17b67ab43"
Oct 29 22:06:12.225771 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.225754    4962 scope.go:122] "RemoveContainer" containerID="52700b1df6d64125547285c51c1e9026f981585a8ffa49ac9fc2ee9e31661c7e"
Oct 29 22:06:12.238646 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.238629    4962 scope.go:122] "RemoveContainer" containerID="df85268bcd864b52d4dc77ca2e2d74d4a2274cda41990c4a866581e17b67ab43"
Oct 29 22:06:12.239098 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:06:12.239076    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df85268bcd864b52d4dc77ca2e2d74d4a2274cda41990c4a866581e17b67ab43\": container with ID starting with df85268bcd864b52d4dc77ca2e2d74d4a2274cda41990c4a866581e17b67ab43 not found: ID does not exist" containerID="df85268bcd864b52d4dc77ca2e2d74d4a2274cda41990c4a866581e17b67ab43"
Oct 29 22:06:12.239177 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.239104    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df85268bcd864b52d4dc77ca2e2d74d4a2274cda41990c4a866581e17b67ab43"} err="failed to get container status \"df85268bcd864b52d4dc77ca2e2d74d4a2274cda41990c4a866581e17b67ab43\": rpc error: code = NotFound desc = could not find container \"df85268bcd864b52d4dc77ca2e2d74d4a2274cda41990c4a866581e17b67ab43\": container with ID starting with df85268bcd864b52d4dc77ca2e2d74d4a2274cda41990c4a866581e17b67ab43 not found: ID does not exist"
Oct 29 22:06:12.239177 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.239121    4962 scope.go:122] "RemoveContainer" containerID="52700b1df6d64125547285c51c1e9026f981585a8ffa49ac9fc2ee9e31661c7e"
Oct 29 22:06:12.239469 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:06:12.239446    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52700b1df6d64125547285c51c1e9026f981585a8ffa49ac9fc2ee9e31661c7e\": container with ID starting with 52700b1df6d64125547285c51c1e9026f981585a8ffa49ac9fc2ee9e31661c7e not found: ID does not exist" containerID="52700b1df6d64125547285c51c1e9026f981585a8ffa49ac9fc2ee9e31661c7e"
Oct 29 22:06:12.239562 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.239474    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52700b1df6d64125547285c51c1e9026f981585a8ffa49ac9fc2ee9e31661c7e"} err="failed to get container status \"52700b1df6d64125547285c51c1e9026f981585a8ffa49ac9fc2ee9e31661c7e\": rpc error: code = NotFound desc = could not find container \"52700b1df6d64125547285c51c1e9026f981585a8ffa49ac9fc2ee9e31661c7e\": container with ID starting with 52700b1df6d64125547285c51c1e9026f981585a8ffa49ac9fc2ee9e31661c7e not found: ID does not exist"
Oct 29 22:06:12.269374 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.269348    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"]
Oct 29 22:06:12.272326 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.272305    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-6df539-openshift-default-d87cc874f-h6v5k"]
Oct 29 22:06:12.668381 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.668352    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk"]
Oct 29 22:06:12.669117 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.669103    4962 cpu_manager.go:414] "containerMap: removing container" podUID="630849f8-61af-40a9-80b9-7f4ae5a4ee6e" containerName="istio-proxy"
Oct 29 22:06:12.669165 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.669118    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="630849f8-61af-40a9-80b9-7f4ae5a4ee6e" containerName="istio-proxy"
Oct 29 22:06:12.669273 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.669263    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="630849f8-61af-40a9-80b9-7f4ae5a4ee6e" containerName="istio-proxy"
Oct 29 22:06:12.674511 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.674493    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk"
Oct 29 22:06:12.693209 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.692954    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-2d53cd-serving-cert\""
Oct 29 22:06:12.696947 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.696924    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk"]
Oct 29 22:06:12.766535 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.766499    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91215c4c-55d2-4472-b58b-5612e1012574-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk\" (UID: \"91215c4c-55d2-4472-b58b-5612e1012574\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk"
Oct 29 22:06:12.766727 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.766547    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6f6z\" (UniqueName: \"kubernetes.io/projected/91215c4c-55d2-4472-b58b-5612e1012574-kube-api-access-p6f6z\") pod \"maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk\" (UID: \"91215c4c-55d2-4472-b58b-5612e1012574\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk"
Oct 29 22:06:12.766727 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.766661    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91215c4c-55d2-4472-b58b-5612e1012574-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk\" (UID: \"91215c4c-55d2-4472-b58b-5612e1012574\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk"
Oct 29 22:06:12.766727 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.766711    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91215c4c-55d2-4472-b58b-5612e1012574-metrics-certs\") pod \"maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk\" (UID: \"91215c4c-55d2-4472-b58b-5612e1012574\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk"
Oct 29 22:06:12.868711 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.868673    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91215c4c-55d2-4472-b58b-5612e1012574-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk\" (UID: \"91215c4c-55d2-4472-b58b-5612e1012574\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk"
Oct 29 22:06:12.868884 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.868726    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6f6z\" (UniqueName: \"kubernetes.io/projected/91215c4c-55d2-4472-b58b-5612e1012574-kube-api-access-p6f6z\") pod \"maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk\" (UID: \"91215c4c-55d2-4472-b58b-5612e1012574\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk"
Oct 29 22:06:12.868884 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.868767    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91215c4c-55d2-4472-b58b-5612e1012574-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk\" (UID: \"91215c4c-55d2-4472-b58b-5612e1012574\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk"
Oct 29 22:06:12.868884 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.868804    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91215c4c-55d2-4472-b58b-5612e1012574-metrics-certs\") pod \"maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk\" (UID: \"91215c4c-55d2-4472-b58b-5612e1012574\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk"
Oct 29 22:06:12.869564 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.869527    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91215c4c-55d2-4472-b58b-5612e1012574-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk\" (UID: \"91215c4c-55d2-4472-b58b-5612e1012574\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk"
Oct 29 22:06:12.871492 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.871464    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91215c4c-55d2-4472-b58b-5612e1012574-metrics-certs\") pod \"maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk\" (UID: \"91215c4c-55d2-4472-b58b-5612e1012574\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk"
Oct 29 22:06:12.871822 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.871801    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91215c4c-55d2-4472-b58b-5612e1012574-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk\" (UID: \"91215c4c-55d2-4472-b58b-5612e1012574\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk"
Oct 29 22:06:12.892619 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.892594    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6f6z\" (UniqueName: \"kubernetes.io/projected/91215c4c-55d2-4472-b58b-5612e1012574-kube-api-access-p6f6z\") pod \"maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk\" (UID: \"91215c4c-55d2-4472-b58b-5612e1012574\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk"
Oct 29 22:06:12.989962 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:12.989897    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk"
Oct 29 22:06:13.219308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:13.133574    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="630849f8-61af-40a9-80b9-7f4ae5a4ee6e" path="/var/lib/kubelet/pods/630849f8-61af-40a9-80b9-7f4ae5a4ee6e/volumes"
Oct 29 22:06:13.718553 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:06:13.718523    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91215c4c_55d2_4472_b58b_5612e1012574.slice/crio-9ccc7cc8b42175ad83b799720155094a508b90a887bed50bb273dcb26f617c3c WatchSource:0}: Error finding container 9ccc7cc8b42175ad83b799720155094a508b90a887bed50bb273dcb26f617c3c: Status 404 returned error can't find the container with id 9ccc7cc8b42175ad83b799720155094a508b90a887bed50bb273dcb26f617c3c
Oct 29 22:06:13.718702 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:13.718582    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk"]
Oct 29 22:06:14.510511 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:14.510475    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk" event={"ID":"91215c4c-55d2-4472-b58b-5612e1012574","Type":"ContainerStarted","Data":"6d78cb9ac9283b9c85c9b62a0c123a5ed11c4af8749237588172206b4a7b4800"}
Oct 29 22:06:14.511051 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:14.510547    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk" event={"ID":"91215c4c-55d2-4472-b58b-5612e1012574","Type":"ContainerStarted","Data":"9ccc7cc8b42175ad83b799720155094a508b90a887bed50bb273dcb26f617c3c"}
Oct 29 22:06:14.511051 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:14.510769    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk"
Oct 29 22:06:14.555481 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:14.555432    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk" podStartSLOduration=2.16808552 podStartE2EDuration="2.555418325s" totalImagesPullingTime="387.332805ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:14.553712889 +0000 UTC m=+1195.626669854" watchObservedRunningTime="2026-10-29 22:06:14.555418325 +0000 UTC m=+1195.628375287"
Oct 29 22:06:15.135313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.128819    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9"]
Oct 29 22:06:15.135313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.130061    4962 cpu_manager.go:414] "containerMap: removing container" podUID="630849f8-61af-40a9-80b9-7f4ae5a4ee6e" containerName="istio-proxy"
Oct 29 22:06:15.135313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.130081    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="630849f8-61af-40a9-80b9-7f4ae5a4ee6e" containerName="istio-proxy"
Oct 29 22:06:15.157315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.154491    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="630849f8-61af-40a9-80b9-7f4ae5a4ee6e" containerName="istio-proxy"
Oct 29 22:06:15.203584 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.189570    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9"]
Oct 29 22:06:15.203584 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.189755    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9"
Oct 29 22:06:15.224864 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.224692    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f363a3-dockercfg-xnb7p\""
Oct 29 22:06:15.241853 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.240570    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed4e4a6-8b44-4368-9194-47aa1bd10f0f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9\" (UID: \"5ed4e4a6-8b44-4368-9194-47aa1bd10f0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9"
Oct 29 22:06:15.241853 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.240924    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6vqn\" (UniqueName: \"kubernetes.io/projected/5ed4e4a6-8b44-4368-9194-47aa1bd10f0f-kube-api-access-m6vqn\") pod \"payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9\" (UID: \"5ed4e4a6-8b44-4368-9194-47aa1bd10f0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9"
Oct 29 22:06:15.241853 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.241601    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f363a3\""
Oct 29 22:06:15.248791 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.248763    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22"]
Oct 29 22:06:15.306243 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.306215    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22"
Oct 29 22:06:15.370311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.348625    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed4e4a6-8b44-4368-9194-47aa1bd10f0f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9\" (UID: \"5ed4e4a6-8b44-4368-9194-47aa1bd10f0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9"
Oct 29 22:06:15.370311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.348986    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6vqn\" (UniqueName: \"kubernetes.io/projected/5ed4e4a6-8b44-4368-9194-47aa1bd10f0f-kube-api-access-m6vqn\") pod \"payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9\" (UID: \"5ed4e4a6-8b44-4368-9194-47aa1bd10f0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9"
Oct 29 22:06:15.370311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.349087    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m47m4\" (UniqueName: \"kubernetes.io/projected/0ae786ab-5df2-457c-bbd9-93d6733cd7c4-kube-api-access-m47m4\") pod \"payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22\" (UID: \"0ae786ab-5df2-457c-bbd9-93d6733cd7c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22"
Oct 29 22:06:15.370311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.349156    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae786ab-5df2-457c-bbd9-93d6733cd7c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22\" (UID: \"0ae786ab-5df2-457c-bbd9-93d6733cd7c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22"
Oct 29 22:06:15.400259 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.397480    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed4e4a6-8b44-4368-9194-47aa1bd10f0f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9\" (UID: \"5ed4e4a6-8b44-4368-9194-47aa1bd10f0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9"
Oct 29 22:06:15.400259 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.397571    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22"]
Oct 29 22:06:15.464181 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.457858    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m47m4\" (UniqueName: \"kubernetes.io/projected/0ae786ab-5df2-457c-bbd9-93d6733cd7c4-kube-api-access-m47m4\") pod \"payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22\" (UID: \"0ae786ab-5df2-457c-bbd9-93d6733cd7c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22"
Oct 29 22:06:15.464570 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.464541    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae786ab-5df2-457c-bbd9-93d6733cd7c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22\" (UID: \"0ae786ab-5df2-457c-bbd9-93d6733cd7c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22"
Oct 29 22:06:15.468134 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.468065    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae786ab-5df2-457c-bbd9-93d6733cd7c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22\" (UID: \"0ae786ab-5df2-457c-bbd9-93d6733cd7c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22"
Oct 29 22:06:15.500432 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.500353    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6vqn\" (UniqueName: \"kubernetes.io/projected/5ed4e4a6-8b44-4368-9194-47aa1bd10f0f-kube-api-access-m6vqn\") pod \"payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9\" (UID: \"5ed4e4a6-8b44-4368-9194-47aa1bd10f0f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9"
Oct 29 22:06:15.625384 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.625346    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9"
Oct 29 22:06:15.642662 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.642630    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m47m4\" (UniqueName: \"kubernetes.io/projected/0ae786ab-5df2-457c-bbd9-93d6733cd7c4-kube-api-access-m47m4\") pod \"payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22\" (UID: \"0ae786ab-5df2-457c-bbd9-93d6733cd7c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22"
Oct 29 22:06:15.653378 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.649473    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s"]
Oct 29 22:06:15.718651 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.715243    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s"]
Oct 29 22:06:15.718651 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.715461    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s"
Oct 29 22:06:15.777862 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.777822    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22"
Oct 29 22:06:15.801650 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.801613    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-301fcc-serving-cert\""
Oct 29 22:06:15.920548 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.920511    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7wm5\" (UniqueName: \"kubernetes.io/projected/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-kube-api-access-d7wm5\") pod \"maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s\" (UID: \"4f572782-bbae-4cb0-a055-b4c9d9e8db5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s"
Oct 29 22:06:15.921045 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.921023    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-maas-api-tls\") pod \"maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s\" (UID: \"4f572782-bbae-4cb0-a055-b4c9d9e8db5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s"
Oct 29 22:06:15.921357 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.921336    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-metrics-certs\") pod \"maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s\" (UID: \"4f572782-bbae-4cb0-a055-b4c9d9e8db5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s"
Oct 29 22:06:15.921501 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:15.921486    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-openshift-service-ca\") pod \"maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s\" (UID: \"4f572782-bbae-4cb0-a055-b4c9d9e8db5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s"
Oct 29 22:06:16.023093 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:16.023058    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-maas-api-tls\") pod \"maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s\" (UID: \"4f572782-bbae-4cb0-a055-b4c9d9e8db5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s"
Oct 29 22:06:16.023232 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:16.023162    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-metrics-certs\") pod \"maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s\" (UID: \"4f572782-bbae-4cb0-a055-b4c9d9e8db5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s"
Oct 29 22:06:16.023232 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:16.023209    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-openshift-service-ca\") pod \"maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s\" (UID: \"4f572782-bbae-4cb0-a055-b4c9d9e8db5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s"
Oct 29 22:06:16.023391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:16.023379    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7wm5\" (UniqueName: \"kubernetes.io/projected/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-kube-api-access-d7wm5\") pod \"maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s\" (UID: \"4f572782-bbae-4cb0-a055-b4c9d9e8db5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s"
Oct 29 22:06:16.023999 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:16.023965    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-openshift-service-ca\") pod \"maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s\" (UID: \"4f572782-bbae-4cb0-a055-b4c9d9e8db5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s"
Oct 29 22:06:16.025782 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:16.025757    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-metrics-certs\") pod \"maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s\" (UID: \"4f572782-bbae-4cb0-a055-b4c9d9e8db5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s"
Oct 29 22:06:16.026081 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:16.026057    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-maas-api-tls\") pod \"maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s\" (UID: \"4f572782-bbae-4cb0-a055-b4c9d9e8db5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s"
Oct 29 22:06:16.051036 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:16.051009    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7wm5\" (UniqueName: \"kubernetes.io/projected/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-kube-api-access-d7wm5\") pod \"maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s\" (UID: \"4f572782-bbae-4cb0-a055-b4c9d9e8db5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s"
Oct 29 22:06:16.196021 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:16.187692    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s"
Oct 29 22:06:16.279480 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:06:16.279443    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed4e4a6_8b44_4368_9194_47aa1bd10f0f.slice/crio-d3a55dd5c2f8dab836a77197f03325bb118efa2430e9e20a0b5cd0cc0b531836 WatchSource:0}: Error finding container d3a55dd5c2f8dab836a77197f03325bb118efa2430e9e20a0b5cd0cc0b531836: Status 404 returned error can't find the container with id d3a55dd5c2f8dab836a77197f03325bb118efa2430e9e20a0b5cd0cc0b531836
Oct 29 22:06:16.279720 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:16.279696    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9"]
Oct 29 22:06:16.308086 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:16.308056    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22"]
Oct 29 22:06:16.702079 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:16.684345    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s"]
Oct 29 22:06:16.758956 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:06:16.758918    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f572782_bbae_4cb0_a055_b4c9d9e8db5a.slice/crio-2c301278bf93dc0a66d2e79a73e557d7e078169a2a050abd6ff3b6bff4f72fd3 WatchSource:0}: Error finding container 2c301278bf93dc0a66d2e79a73e557d7e078169a2a050abd6ff3b6bff4f72fd3: Status 404 returned error can't find the container with id 2c301278bf93dc0a66d2e79a73e557d7e078169a2a050abd6ff3b6bff4f72fd3
Oct 29 22:06:16.760927 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:16.760239    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9" event={"ID":"5ed4e4a6-8b44-4368-9194-47aa1bd10f0f","Type":"ContainerStarted","Data":"4b1239a6dabc339cdef95c47598a1a68520e824235fdb8f89c4c2133849bcf53"}
Oct 29 22:06:16.760927 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:16.760380    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9" event={"ID":"5ed4e4a6-8b44-4368-9194-47aa1bd10f0f","Type":"ContainerStarted","Data":"d3a55dd5c2f8dab836a77197f03325bb118efa2430e9e20a0b5cd0cc0b531836"}
Oct 29 22:06:16.760927 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:16.760444    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9"
Oct 29 22:06:16.775841 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:16.775801    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22" event={"ID":"0ae786ab-5df2-457c-bbd9-93d6733cd7c4","Type":"ContainerStarted","Data":"ce340fc13f2c9634d1e6aa1cc5ad1defa0facaa94ee03760120409d3e6df913f"}
Oct 29 22:06:16.775903 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:16.775857    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22" event={"ID":"0ae786ab-5df2-457c-bbd9-93d6733cd7c4","Type":"ContainerStarted","Data":"d4f7a1dd6172b3ad084efe57db4e0b76715c46a942b057f1c318e56e9293814f"}
Oct 29 22:06:16.776950 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:16.776924    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22"
Oct 29 22:06:16.826919 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:16.826876    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9" podStartSLOduration=1.826862218 podStartE2EDuration="1.826862218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:16.822819729 +0000 UTC m=+1197.895776693" watchObservedRunningTime="2026-10-29 22:06:16.826862218 +0000 UTC m=+1197.899819197"
Oct 29 22:06:16.895385 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:16.895326    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22" podStartSLOduration=1.895310794 podStartE2EDuration="1.895310794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:16.892246087 +0000 UTC m=+1197.965203051" watchObservedRunningTime="2026-10-29 22:06:16.895310794 +0000 UTC m=+1197.968267765"
Oct 29 22:06:17.789459 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:17.789411    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s" event={"ID":"4f572782-bbae-4cb0-a055-b4c9d9e8db5a","Type":"ContainerStarted","Data":"55c5e0f5c9409b457b573ed94923e01ad880bc0597331faf94a45dd43b747867"}
Oct 29 22:06:17.789937 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:17.789473    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s" event={"ID":"4f572782-bbae-4cb0-a055-b4c9d9e8db5a","Type":"ContainerStarted","Data":"2c301278bf93dc0a66d2e79a73e557d7e078169a2a050abd6ff3b6bff4f72fd3"}
Oct 29 22:06:17.790003 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:17.789971    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s"
Oct 29 22:06:17.838530 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:17.838484    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s" podStartSLOduration=2.450321933 podStartE2EDuration="2.838473093s" totalImagesPullingTime="388.15116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:17.837197979 +0000 UTC m=+1198.910154945" watchObservedRunningTime="2026-10-29 22:06:17.838473093 +0000 UTC m=+1198.911430058"
Oct 29 22:06:19.109183 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:19.109154    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz"]
Oct 29 22:06:19.594079 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:19.594052    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/2.log"
Oct 29 22:06:19.596761 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:19.596739    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/2.log"
Oct 29 22:06:19.601574 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:19.601553    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74bdq_259e6b95-d64c-4101-871f-4a933d4ac38f/ovn-acl-logging/0.log"
Oct 29 22:06:19.604078 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:19.604059    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74bdq_259e6b95-d64c-4101-871f-4a933d4ac38f/ovn-acl-logging/0.log"
Oct 29 22:06:20.528928 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:20.528900    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk"
Oct 29 22:06:23.803835 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:23.803807    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s"
Oct 29 22:06:24.110474 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:24.110391    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz" podUID="8cd29dac-756d-4875-8fa3-3f9662b139e2" containerName="payload-pre-processing" containerID="cri-o://07fce3c11363df39133e430a34603873e6b06a706d635f34a541e1bd23d2c1f9" gracePeriod=25
Oct 29 22:06:24.775844 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:24.775814    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz"
Oct 29 22:06:24.841104 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:24.841065    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cd29dac-756d-4875-8fa3-3f9662b139e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd29dac-756d-4875-8fa3-3f9662b139e2-plugins-config-volume\") pod \"8cd29dac-756d-4875-8fa3-3f9662b139e2\" (UID: \"8cd29dac-756d-4875-8fa3-3f9662b139e2\") "
Oct 29 22:06:24.841587 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:24.841473    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cd29dac-756d-4875-8fa3-3f9662b139e2-kube-api-access-6n9qn\" (UniqueName: \"kubernetes.io/projected/8cd29dac-756d-4875-8fa3-3f9662b139e2-kube-api-access-6n9qn\") pod \"8cd29dac-756d-4875-8fa3-3f9662b139e2\" (UID: \"8cd29dac-756d-4875-8fa3-3f9662b139e2\") "
Oct 29 22:06:24.841587 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:24.841481    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cd29dac-756d-4875-8fa3-3f9662b139e2-plugins-config-volume" pod "8cd29dac-756d-4875-8fa3-3f9662b139e2" (UID: "8cd29dac-756d-4875-8fa3-3f9662b139e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:06:24.842130 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:24.842110    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd29dac-756d-4875-8fa3-3f9662b139e2-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:06:24.843614 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:24.843586    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd29dac-756d-4875-8fa3-3f9662b139e2-kube-api-access-6n9qn" pod "8cd29dac-756d-4875-8fa3-3f9662b139e2" (UID: "8cd29dac-756d-4875-8fa3-3f9662b139e2"). InnerVolumeSpecName "kube-api-access-6n9qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:06:24.912554 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:24.912484    4962 generic.go:356] "Generic (PLEG): container finished" podID="8cd29dac-756d-4875-8fa3-3f9662b139e2" containerID="07fce3c11363df39133e430a34603873e6b06a706d635f34a541e1bd23d2c1f9" exitCode=0
Oct 29 22:06:24.912554 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:24.912519    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz" event={"ID":"8cd29dac-756d-4875-8fa3-3f9662b139e2","Type":"ContainerDied","Data":"07fce3c11363df39133e430a34603873e6b06a706d635f34a541e1bd23d2c1f9"}
Oct 29 22:06:24.912554 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:24.912537    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz"
Oct 29 22:06:24.912754 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:24.912575    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz" event={"ID":"8cd29dac-756d-4875-8fa3-3f9662b139e2","Type":"ContainerDied","Data":"4c7209e54be72a2c94a3d74aec058266a4ecf727b6fa203d00fcc142d54a8d4f"}
Oct 29 22:06:24.912754 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:24.912598    4962 scope.go:122] "RemoveContainer" containerID="07fce3c11363df39133e430a34603873e6b06a706d635f34a541e1bd23d2c1f9"
Oct 29 22:06:24.926704 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:24.926682    4962 scope.go:122] "RemoveContainer" containerID="07fce3c11363df39133e430a34603873e6b06a706d635f34a541e1bd23d2c1f9"
Oct 29 22:06:24.926980 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:06:24.926962    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07fce3c11363df39133e430a34603873e6b06a706d635f34a541e1bd23d2c1f9\": container with ID starting with 07fce3c11363df39133e430a34603873e6b06a706d635f34a541e1bd23d2c1f9 not found: ID does not exist" containerID="07fce3c11363df39133e430a34603873e6b06a706d635f34a541e1bd23d2c1f9"
Oct 29 22:06:24.927032 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:24.926989    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fce3c11363df39133e430a34603873e6b06a706d635f34a541e1bd23d2c1f9"} err="failed to get container status \"07fce3c11363df39133e430a34603873e6b06a706d635f34a541e1bd23d2c1f9\": rpc error: code = NotFound desc = could not find container \"07fce3c11363df39133e430a34603873e6b06a706d635f34a541e1bd23d2c1f9\": container with ID starting with 07fce3c11363df39133e430a34603873e6b06a706d635f34a541e1bd23d2c1f9 not found: ID does not exist"
Oct 29 22:06:24.943677 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:24.943653    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n9qn\" (UniqueName: \"kubernetes.io/projected/8cd29dac-756d-4875-8fa3-3f9662b139e2-kube-api-access-6n9qn\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:06:24.968347 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:24.968327    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz"]
Oct 29 22:06:24.972408 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:24.972388    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-751a7416-9bcfd9775-q9wfz"]
Oct 29 22:06:25.061133 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:25.061109    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cd29dac-756d-4875-8fa3-3f9662b139e2" path="/var/lib/kubelet/pods/8cd29dac-756d-4875-8fa3-3f9662b139e2/volumes"
Oct 29 22:06:27.792488 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:27.792402    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9"
Oct 29 22:06:28.800968 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:28.800936    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22"
Oct 29 22:06:39.722410 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:06:39.722312    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:06:42.305633 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:42.305592    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d"]
Oct 29 22:06:42.306453 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:42.306436    4962 cpu_manager.go:414] "containerMap: removing container" podUID="8cd29dac-756d-4875-8fa3-3f9662b139e2" containerName="payload-pre-processing"
Oct 29 22:06:42.306511 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:42.306453    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cd29dac-756d-4875-8fa3-3f9662b139e2" containerName="payload-pre-processing"
Oct 29 22:06:42.306573 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:42.306563    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cd29dac-756d-4875-8fa3-3f9662b139e2" containerName="payload-pre-processing"
Oct 29 22:06:42.310464 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:42.310446    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d"
Oct 29 22:06:42.325984 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:42.325961    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f363a3-serving-cert\""
Oct 29 22:06:42.343009 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:42.342981    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d"]
Oct 29 22:06:42.538310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:42.535698    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf2ebea-429f-41f3-9234-27a2df85ddf7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d\" (UID: \"caf2ebea-429f-41f3-9234-27a2df85ddf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d"
Oct 29 22:06:42.538310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:42.535913    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caf2ebea-429f-41f3-9234-27a2df85ddf7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d\" (UID: \"caf2ebea-429f-41f3-9234-27a2df85ddf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d"
Oct 29 22:06:42.538310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:42.535975    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf2ebea-429f-41f3-9234-27a2df85ddf7-metrics-certs\") pod \"maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d\" (UID: \"caf2ebea-429f-41f3-9234-27a2df85ddf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d"
Oct 29 22:06:42.538310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:42.536067    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdmzk\" (UniqueName: \"kubernetes.io/projected/caf2ebea-429f-41f3-9234-27a2df85ddf7-kube-api-access-vdmzk\") pod \"maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d\" (UID: \"caf2ebea-429f-41f3-9234-27a2df85ddf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d"
Oct 29 22:06:42.638207 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:42.638067    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf2ebea-429f-41f3-9234-27a2df85ddf7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d\" (UID: \"caf2ebea-429f-41f3-9234-27a2df85ddf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d"
Oct 29 22:06:42.642736 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:42.638648    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caf2ebea-429f-41f3-9234-27a2df85ddf7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d\" (UID: \"caf2ebea-429f-41f3-9234-27a2df85ddf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d"
Oct 29 22:06:42.649774 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:42.649738    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf2ebea-429f-41f3-9234-27a2df85ddf7-metrics-certs\") pod \"maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d\" (UID: \"caf2ebea-429f-41f3-9234-27a2df85ddf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d"
Oct 29 22:06:42.650113 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:42.650086    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdmzk\" (UniqueName: \"kubernetes.io/projected/caf2ebea-429f-41f3-9234-27a2df85ddf7-kube-api-access-vdmzk\") pod \"maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d\" (UID: \"caf2ebea-429f-41f3-9234-27a2df85ddf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d"
Oct 29 22:06:42.650976 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:42.649205    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf2ebea-429f-41f3-9234-27a2df85ddf7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d\" (UID: \"caf2ebea-429f-41f3-9234-27a2df85ddf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d"
Oct 29 22:06:42.658350 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:42.658170    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf2ebea-429f-41f3-9234-27a2df85ddf7-metrics-certs\") pod \"maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d\" (UID: \"caf2ebea-429f-41f3-9234-27a2df85ddf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d"
Oct 29 22:06:42.660237 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:42.660209    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caf2ebea-429f-41f3-9234-27a2df85ddf7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d\" (UID: \"caf2ebea-429f-41f3-9234-27a2df85ddf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d"
Oct 29 22:06:42.681010 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:42.680979    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdmzk\" (UniqueName: \"kubernetes.io/projected/caf2ebea-429f-41f3-9234-27a2df85ddf7-kube-api-access-vdmzk\") pod \"maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d\" (UID: \"caf2ebea-429f-41f3-9234-27a2df85ddf7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d"
Oct 29 22:06:42.927176 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:42.927144    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d"
Oct 29 22:06:43.333095 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:43.333066    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d"]
Oct 29 22:06:43.339751 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:43.339721    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d" event={"ID":"caf2ebea-429f-41f3-9234-27a2df85ddf7","Type":"ContainerStarted","Data":"8983afb15d532efb839896659ae10a840d8aa304e532cf836fa55e1350d1ed17"}
Oct 29 22:06:44.408315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:44.405680    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d" event={"ID":"caf2ebea-429f-41f3-9234-27a2df85ddf7","Type":"ContainerStarted","Data":"986105593c74aa4a9fd28589e110130cbf8a961d32220b53133c220d96a7fe77"}
Oct 29 22:06:44.408315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:44.407042    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d"
Oct 29 22:06:44.483685 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:44.483623    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d" podStartSLOduration=1.923005715 podStartE2EDuration="2.483605396s" totalImagesPullingTime="560.599681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:06:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:06:44.483200268 +0000 UTC m=+1225.556157254" watchObservedRunningTime="2026-10-29 22:06:44.483605396 +0000 UTC m=+1225.556562363"
Oct 29 22:06:50.261204 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:50.261166    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw"]
Oct 29 22:06:50.325038 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:50.324329    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c"]
Oct 29 22:06:51.712265 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:51.712217    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:52.002843 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:52.002674    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:06:52.074326 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:52.063596    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:52.087321 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:52.075412    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:06:52.226944 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:52.226905    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj"]
Oct 29 22:06:52.332225 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:52.332112    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:52.692421 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:52.692380    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded"
Oct 29 22:06:52.863897 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:52.863851    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:06:52.883797 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:52.881832    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:06:52.930915 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:52.930874    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d"
Oct 29 22:06:52.941559 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:52.941508    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:52.970368 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:52.955722    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:53.014462 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:53.014421    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:06:53.016381 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:53.015222    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 22:06:53.031772 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:53.031735    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:53.384361 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:53.032043    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:53.551374 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:53.085385    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:53.551374 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:53.103631    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:53.551374 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:53.404497    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:06:53.551374 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:53.463462    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:06:53.551374 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:53.463533    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 22:06:53.551374 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:53.463573    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:06:53.551374 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:53.463611    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 22:06:53.869438 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:53.869390    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:53.897310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:53.875197    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 22:06:54.087393 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:54.022024    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:54.323013 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:54.322969    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded"
Oct 29 22:06:54.441315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:54.425574    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:54.453312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:54.449913    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:06:54.535316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:54.498160    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:06:54.539665 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:54.498265    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:54.603989 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:54.498211    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:06:54.660426 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:54.660383    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:54.666340 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:54.660248    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:54.681607 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:54.539073    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:54.682418 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:54.682370    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:55.290392 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:55.229866    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:55.290392 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:55.231093    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:06:55.324321 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:55.316013    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw" podUID="2d397882-c29a-47e6-a6d5-280f529d89d6" containerName="payload-processing" containerID="cri-o://cd9c4448c006a9ad263a8214bb9014d1e12c3322403257fa5d443ceebc769f95" gracePeriod=25
Oct 29 22:06:55.399309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:55.373580    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c" podUID="263a0c66-6465-4269-b15c-28ccb8afbe32" containerName="payload-pre-processing" containerID="cri-o://6d358f03a7fd07efe18ceb64b79c3b1677ae19dd4b5043b9f5332fc5f2dc1533" gracePeriod=25
Oct 29 22:06:55.486951 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:55.479408    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:06:55.568555 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:55.554665    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:55.568555 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:55.566589    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:55.671314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:55.670830    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:55.712316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:55.706326    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:06:55.743345 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:55.723838    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:06:55.743345 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:55.735543    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:06:55.770549 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:55.770507    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:06:55.811912 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:55.796467    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 22:06:55.812324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:55.812305    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:06:55.812531 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:55.812509    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:06:56.089312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:56.080652    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.252513 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:06:56.245779    4962 logging.go:55] [core] [Channel #2305 SubChannel #2307]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.115:50052", ServerName: "10.132.0.115:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.115:50052: connect: connection refused"
Oct 29 22:06:56.285318 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:06:56.280515    4962 logging.go:55] [core] [Channel #2304 SubChannel #2306]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.116:50052", ServerName: "10.132.0.116:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.116:50052: connect: connection refused"
Oct 29 22:06:56.335341 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:56.330127    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:06:56.377312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:56.370917    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:06:56.511821 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:56.487124    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.634313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:56.631663    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:06:56.634313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:56.633120    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.675733 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:56.675687    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.717529 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:56.717493    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:06:56.718605 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:56.718573    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds" podUID="2a71a069-94f9-4736-8b9f-d43292554254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.746091 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:56.743452    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.747424 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:56.743548    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.813487 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:56.813381    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:56.937760 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:56.717493    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:57.292407 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:57.292348    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c" podUID="263a0c66-6465-4269-b15c-28ccb8afbe32" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.115:50052\" within 1s: context deadline exceeded"
Oct 29 22:06:57.394523 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:57.394450    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:06:57.496542 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:57.396462    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw" podUID="2d397882-c29a-47e6-a6d5-280f529d89d6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.116:50052\" within 1s: context deadline exceeded"
Oct 29 22:06:57.496856 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:57.400966    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:06:57.627644 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:57.627533    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:06:57.684395 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:57.628585    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:06:57.684395 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:57.668330    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg" podUID="16540bea-446b-46d9-9cde-4c9c34785ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:57.717312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:57.687110    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:57.785321 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:57.781951    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj" podUID="45b4cce5-7740-4811-9191-9bfddb5e3fac" containerName="payload-pre-processing" containerID="cri-o://9ed2aac9a09967bf8a8e9d07727c9efb6d18b181612d36ad7f0560308fcde426" gracePeriod=25
Oct 29 22:06:57.801319 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:57.797493    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:57.965976 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:57.965935    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded"
Oct 29 22:06:58.076825 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:57.857768    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:58.079216 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:58.079177    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:58.514474 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:58.485726    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:58.514474 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:58.502450    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:58.651236 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:58.632415    4962 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-jdps8 container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.17:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:06:58.651236 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:58.632519    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-jdps8" podUID="f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.17:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:58.679655 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:58.679597    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:06:58.688492 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:58.688429    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:58.931702 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:58.931670    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:06:59.217350 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:59.217235    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:06:59.217894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:59.217863    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:59.218281 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:58.931623    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:06:59.262820 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:59.168489    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:59.262820 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:58.931954    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:59.262820 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:59.027985    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:06:59.449312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:59.399162    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw"
Oct 29 22:06:59.512432 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:59.417747    4962 generic.go:356] "Generic (PLEG): container finished" podID="2d397882-c29a-47e6-a6d5-280f529d89d6" containerID="cd9c4448c006a9ad263a8214bb9014d1e12c3322403257fa5d443ceebc769f95" exitCode=0
Oct 29 22:06:59.520309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:59.518469    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d" podUID="696bd863-08fa-4694-843f-ab65f3ccfeed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:59.543309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:59.520766    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:59.549109 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:06:59.549074    4962 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.427s"
Oct 29 22:06:59.595429 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:59.577406    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:59.606043 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:59.606007    4962 generic.go:356] "Generic (PLEG): container finished" podID="263a0c66-6465-4269-b15c-28ccb8afbe32" containerID="6d358f03a7fd07efe18ceb64b79c3b1677ae19dd4b5043b9f5332fc5f2dc1533" exitCode=0
Oct 29 22:06:59.629397 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:59.629347    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d397882-c29a-47e6-a6d5-280f529d89d6-kube-api-access-v2sk4\" (UniqueName: \"kubernetes.io/projected/2d397882-c29a-47e6-a6d5-280f529d89d6-kube-api-access-v2sk4\") pod \"2d397882-c29a-47e6-a6d5-280f529d89d6\" (UID: \"2d397882-c29a-47e6-a6d5-280f529d89d6\") "
Oct 29 22:06:59.629736 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:59.629695    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d397882-c29a-47e6-a6d5-280f529d89d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d397882-c29a-47e6-a6d5-280f529d89d6-plugins-config-volume\") pod \"2d397882-c29a-47e6-a6d5-280f529d89d6\" (UID: \"2d397882-c29a-47e6-a6d5-280f529d89d6\") "
Oct 29 22:06:59.631572 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:59.631544    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d397882-c29a-47e6-a6d5-280f529d89d6-plugins-config-volume" pod "2d397882-c29a-47e6-a6d5-280f529d89d6" (UID: "2d397882-c29a-47e6-a6d5-280f529d89d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:06:59.700397 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:59.681458    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:59.726784 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:59.719555    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:06:59.926362 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:06:59.745917    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d397882-c29a-47e6-a6d5-280f529d89d6-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:07:00.241308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.215484    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.241308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.224196    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.241308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.225458    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:07:00.260414 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.256754    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.260414 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.257661    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:07:00.260414 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.257943    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:07:00.271183 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.271138    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw" event={"ID":"2d397882-c29a-47e6-a6d5-280f529d89d6","Type":"ContainerDied","Data":"cd9c4448c006a9ad263a8214bb9014d1e12c3322403257fa5d443ceebc769f95"}
Oct 29 22:07:00.271499 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.271460    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c" event={"ID":"263a0c66-6465-4269-b15c-28ccb8afbe32","Type":"ContainerDied","Data":"6d358f03a7fd07efe18ceb64b79c3b1677ae19dd4b5043b9f5332fc5f2dc1533"}
Oct 29 22:07:00.293327 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.272973    4962 scope.go:122] "RemoveContainer" containerID="cd9c4448c006a9ad263a8214bb9014d1e12c3322403257fa5d443ceebc769f95"
Oct 29 22:07:00.300070 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.300036    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d397882-c29a-47e6-a6d5-280f529d89d6-kube-api-access-v2sk4" pod "2d397882-c29a-47e6-a6d5-280f529d89d6" (UID: "2d397882-c29a-47e6-a6d5-280f529d89d6"). InnerVolumeSpecName "kube-api-access-v2sk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:07:00.348524 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.348499    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c"
Oct 29 22:07:00.431010 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.430921    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2sk4\" (UniqueName: \"kubernetes.io/projected/2d397882-c29a-47e6-a6d5-280f529d89d6-kube-api-access-v2sk4\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:07:00.522497 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.522449    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:00.523127 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.523078    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57" podUID="67b4a685-bfa5-4a36-adf7-68c7591137c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.523790 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.523765    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.536370 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.536331    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.572144 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:07:00.552552    4962 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d397882_c29a_47e6_a6d5_280f529d89d6.slice/crio-conmon-cd9c4448c006a9ad263a8214bb9014d1e12c3322403257fa5d443ceebc769f95.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d397882_c29a_47e6_a6d5_280f529d89d6.slice/crio-cd9c4448c006a9ad263a8214bb9014d1e12c3322403257fa5d443ceebc769f95.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b4cce5_7740_4811_9191_9bfddb5e3fac.slice/crio-conmon-9ed2aac9a09967bf8a8e9d07727c9efb6d18b181612d36ad7f0560308fcde426.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263a0c66_6465_4269_b15c_28ccb8afbe32.slice/crio-conmon-6d358f03a7fd07efe18ceb64b79c3b1677ae19dd4b5043b9f5332fc5f2dc1533.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263a0c66_6465_4269_b15c_28ccb8afbe32.slice/crio-6d358f03a7fd07efe18ceb64b79c3b1677ae19dd4b5043b9f5332fc5f2dc1533.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b4cce5_7740_4811_9191_9bfddb5e3fac.slice/crio-9ed2aac9a09967bf8a8e9d07727c9efb6d18b181612d36ad7f0560308fcde426.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:07:00.580238 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.569869    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/263a0c66-6465-4269-b15c-28ccb8afbe32-kube-api-access-ksk95\" (UniqueName: \"kubernetes.io/projected/263a0c66-6465-4269-b15c-28ccb8afbe32-kube-api-access-ksk95\") pod \"263a0c66-6465-4269-b15c-28ccb8afbe32\" (UID: \"263a0c66-6465-4269-b15c-28ccb8afbe32\") "
Oct 29 22:07:00.580732 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.580697    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/263a0c66-6465-4269-b15c-28ccb8afbe32-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263a0c66-6465-4269-b15c-28ccb8afbe32-plugins-config-volume\") pod \"263a0c66-6465-4269-b15c-28ccb8afbe32\" (UID: \"263a0c66-6465-4269-b15c-28ccb8afbe32\") "
Oct 29 22:07:00.592471 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.579275    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.601035 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.600997    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/263a0c66-6465-4269-b15c-28ccb8afbe32-plugins-config-volume" pod "263a0c66-6465-4269-b15c-28ccb8afbe32" (UID: "263a0c66-6465-4269-b15c-28ccb8afbe32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:07:00.601923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.601891    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263a0c66-6465-4269-b15c-28ccb8afbe32-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:07:00.602167 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.602147    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:00.637600 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.637545    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp" podUID="5d5e71cf-5078-4917-94b8-2b531b143393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.642748 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.642643    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.712694 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.712596    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263a0c66-6465-4269-b15c-28ccb8afbe32-kube-api-access-ksk95" pod "263a0c66-6465-4269-b15c-28ccb8afbe32" (UID: "263a0c66-6465-4269-b15c-28ccb8afbe32"). InnerVolumeSpecName "kube-api-access-ksk95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:07:00.732495 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.715280    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.762592 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:07:00.762554    4962 logging.go:55] [core] [Channel #2319 SubChannel #2320]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.122:50052", ServerName: "10.132.0.122:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.122:50052: connect: connection refused"
Oct 29 22:07:00.771847 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.762907    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:07:00.883606 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.883570    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw" event={"ID":"2d397882-c29a-47e6-a6d5-280f529d89d6","Type":"ContainerDied","Data":"3a797f17a9669a5b41770d14f76e46b3feb843c0607ae18e2cddddac94f8d190"}
Oct 29 22:07:00.910894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.885066    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:00.911327 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.893349    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksk95\" (UniqueName: \"kubernetes.io/projected/263a0c66-6465-4269-b15c-28ccb8afbe32-kube-api-access-ksk95\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:07:00.911515 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:00.894511    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw"
Oct 29 22:07:01.362840 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:01.211752    4962 scope.go:122] "RemoveContainer" containerID="cd9c4448c006a9ad263a8214bb9014d1e12c3322403257fa5d443ceebc769f95"
Oct 29 22:07:01.362840 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:07:01.256369    4962 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b4cce5_7740_4811_9191_9bfddb5e3fac.slice/crio-conmon-9ed2aac9a09967bf8a8e9d07727c9efb6d18b181612d36ad7f0560308fcde426.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b4cce5_7740_4811_9191_9bfddb5e3fac.slice/crio-9ed2aac9a09967bf8a8e9d07727c9efb6d18b181612d36ad7f0560308fcde426.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:07:01.362840 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:01.256187    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:01.362840 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:07:01.257847    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd9c4448c006a9ad263a8214bb9014d1e12c3322403257fa5d443ceebc769f95\": container with ID starting with cd9c4448c006a9ad263a8214bb9014d1e12c3322403257fa5d443ceebc769f95 not found: ID does not exist" containerID="cd9c4448c006a9ad263a8214bb9014d1e12c3322403257fa5d443ceebc769f95"
Oct 29 22:07:01.362840 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:01.257884    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd9c4448c006a9ad263a8214bb9014d1e12c3322403257fa5d443ceebc769f95"} err="failed to get container status \"cd9c4448c006a9ad263a8214bb9014d1e12c3322403257fa5d443ceebc769f95\": rpc error: code = NotFound desc = could not find container \"cd9c4448c006a9ad263a8214bb9014d1e12c3322403257fa5d443ceebc769f95\": container with ID starting with cd9c4448c006a9ad263a8214bb9014d1e12c3322403257fa5d443ceebc769f95 not found: ID does not exist"
Oct 29 22:07:01.362840 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:01.318246    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:07:01.362840 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:01.323648    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:01.362840 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:01.334887    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:01.362840 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:01.335786    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:07:01.362840 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:01.354712    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:01.498906 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:01.469883    4962 generic.go:356] "Generic (PLEG): container finished" podID="45b4cce5-7740-4811-9191-9bfddb5e3fac" containerID="9ed2aac9a09967bf8a8e9d07727c9efb6d18b181612d36ad7f0560308fcde426" exitCode=0
Oct 29 22:07:01.537872 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:01.537116    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:01.603189 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:01.603146    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:07:01.735467 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:01.733717    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:01.854596 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:01.844245    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj" podUID="45b4cce5-7740-4811-9191-9bfddb5e3fac" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.122:50052\" within 1s: context deadline exceeded"
Oct 29 22:07:02.136965 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:02.091959    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj" event={"ID":"45b4cce5-7740-4811-9191-9bfddb5e3fac","Type":"ContainerDied","Data":"9ed2aac9a09967bf8a8e9d07727c9efb6d18b181612d36ad7f0560308fcde426"}
Oct 29 22:07:02.218341 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:02.188403    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:07:02.218341 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:02.189484    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c" event={"ID":"263a0c66-6465-4269-b15c-28ccb8afbe32","Type":"ContainerDied","Data":"0f41381aa652d9d810994594bb75f088c46dae6b9d64cf532c9f5fa62d6b716d"}
Oct 29 22:07:02.218341 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:02.190930    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c"
Oct 29 22:07:02.218341 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:02.191338    4962 scope.go:122] "RemoveContainer" containerID="6d358f03a7fd07efe18ceb64b79c3b1677ae19dd4b5043b9f5332fc5f2dc1533"
Oct 29 22:07:02.258307 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:02.223807    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:02.258307 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:02.223872    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6" podUID="295e3d96-ff42-4be3-b4de-cd85ca4d8578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:02.274322 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:02.273814    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded"
Oct 29 22:07:02.440455 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:02.440408    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:07:02.791845 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:02.757878    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:07:02.887325 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:02.847175    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:07:03.033365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:02.862329    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded"
Oct 29 22:07:03.033365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:02.888947    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847"]
Oct 29 22:07:03.059339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:02.889143    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:07:03.059339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:02.971214    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.059339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:03.038399    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:07:03.059339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:03.040568    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.130325 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:03.041561    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:07:03.130856 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:03.130817    4962 cpu_manager.go:414] "containerMap: removing container" podUID="2d397882-c29a-47e6-a6d5-280f529d89d6" containerName="payload-processing"
Oct 29 22:07:03.130972 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:03.130859    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d397882-c29a-47e6-a6d5-280f529d89d6" containerName="payload-processing"
Oct 29 22:07:03.130972 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:03.130930    4962 cpu_manager.go:414] "containerMap: removing container" podUID="263a0c66-6465-4269-b15c-28ccb8afbe32" containerName="payload-pre-processing"
Oct 29 22:07:03.130972 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:03.130941    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="263a0c66-6465-4269-b15c-28ccb8afbe32" containerName="payload-pre-processing"
Oct 29 22:07:03.149223 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:03.149179    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d397882-c29a-47e6-a6d5-280f529d89d6" containerName="payload-processing"
Oct 29 22:07:03.220802 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:03.220767    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="263a0c66-6465-4269-b15c-28ccb8afbe32" containerName="payload-pre-processing"
Oct 29 22:07:03.321695 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:03.321596    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:03.321984 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:03.321949    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z" podUID="132ed775-c9e6-41f1-b224-fff550c6e073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.322468 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:03.322442    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:03.322623 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:03.322602    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j" podUID="590ad9cd-3237-4783-8dea-82e67274b580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.410206 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:03.203979    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.410206 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:03.396824    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:03.410206 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:03.396879    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2" podUID="03b2ca39-a1c7-45bf-8458-f23215d334b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.695495 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:03.695451    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg" podUID="70e1bafe-4b93-44d7-a179-44d5a82fd069" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": context deadline exceeded"
Oct 29 22:07:03.731533 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:03.731143    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847"]
Oct 29 22:07:03.731533 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:03.731221    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n"]
Oct 29 22:07:03.763525 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:03.760643    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:03.763525 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:03.761142    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847"
Oct 29 22:07:03.884434 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:03.836512    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:04.143519 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:04.143413    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:07:04.276311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:04.251640    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:04.329315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:04.276908    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:04.515693 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:04.515277    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:07:04.544181 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:04.544129    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:04.567538 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:04.276999    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:04.569626 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:04.361030    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:07:04.578840 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:04.379245    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-edc2d3-dockercfg-hghbs\""
Oct 29 22:07:04.629314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:04.430767    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-edc2d3\""
Oct 29 22:07:04.660253 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:04.404399    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:07:04.766479 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:04.276974    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:04.787767 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:04.623484    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj"
Oct 29 22:07:04.788254 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:04.657752    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": context deadline exceeded"
Oct 29 22:07:04.788580 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:04.658532    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6fe16e8-3928-4766-9daf-7a449c61233d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847\" (UID: \"e6fe16e8-3928-4766-9daf-7a449c61233d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847"
Oct 29 22:07:04.801507 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:04.792090    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:04.842690 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:04.842657    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n"]
Oct 29 22:07:04.843246 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:04.843214    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x72r\" (UniqueName: \"kubernetes.io/projected/e6fe16e8-3928-4766-9daf-7a449c61233d-kube-api-access-7x72r\") pod \"payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847\" (UID: \"e6fe16e8-3928-4766-9daf-7a449c61233d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847"
Oct 29 22:07:04.862654 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:04.862436    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:07:04.864825 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:04.864801    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n"
Oct 29 22:07:05.089171 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.088911    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45b4cce5-7740-4811-9191-9bfddb5e3fac-kube-api-access-rb8m2\" (UniqueName: \"kubernetes.io/projected/45b4cce5-7740-4811-9191-9bfddb5e3fac-kube-api-access-rb8m2\") pod \"45b4cce5-7740-4811-9191-9bfddb5e3fac\" (UID: \"45b4cce5-7740-4811-9191-9bfddb5e3fac\") "
Oct 29 22:07:05.089171 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.089009    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45b4cce5-7740-4811-9191-9bfddb5e3fac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b4cce5-7740-4811-9191-9bfddb5e3fac-plugins-config-volume\") pod \"45b4cce5-7740-4811-9191-9bfddb5e3fac\" (UID: \"45b4cce5-7740-4811-9191-9bfddb5e3fac\") "
Oct 29 22:07:05.089738 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.089654    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f96700-2d4a-4b30-ada0-04cf70b7f6d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n\" (UID: \"c2f96700-2d4a-4b30-ada0-04cf70b7f6d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n"
Oct 29 22:07:05.093309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.089943    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8lrn\" (UniqueName: \"kubernetes.io/projected/c2f96700-2d4a-4b30-ada0-04cf70b7f6d2-kube-api-access-x8lrn\") pod \"payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n\" (UID: \"c2f96700-2d4a-4b30-ada0-04cf70b7f6d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n"
Oct 29 22:07:05.093309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.089994    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6fe16e8-3928-4766-9daf-7a449c61233d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847\" (UID: \"e6fe16e8-3928-4766-9daf-7a449c61233d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847"
Oct 29 22:07:05.093309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.090266    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x72r\" (UniqueName: \"kubernetes.io/projected/e6fe16e8-3928-4766-9daf-7a449c61233d-kube-api-access-7x72r\") pod \"payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847\" (UID: \"e6fe16e8-3928-4766-9daf-7a449c61233d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847"
Oct 29 22:07:05.171542 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.145350    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x72r\" (UniqueName: \"kubernetes.io/projected/e6fe16e8-3928-4766-9daf-7a449c61233d-kube-api-access-7x72r\") pod \"payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847\" (UID: \"e6fe16e8-3928-4766-9daf-7a449c61233d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847"
Oct 29 22:07:05.218572 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.200927    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6fe16e8-3928-4766-9daf-7a449c61233d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847\" (UID: \"e6fe16e8-3928-4766-9daf-7a449c61233d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847"
Oct 29 22:07:05.218572 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.202450    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847"
Oct 29 22:07:05.218572 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.212622    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:05.218572 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.212669    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt" podUID="730edf47-8b07-477a-acfc-47faaaacfc84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.218572 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.213928    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8lrn\" (UniqueName: \"kubernetes.io/projected/c2f96700-2d4a-4b30-ada0-04cf70b7f6d2-kube-api-access-x8lrn\") pod \"payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n\" (UID: \"c2f96700-2d4a-4b30-ada0-04cf70b7f6d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n"
Oct 29 22:07:05.218572 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.214145    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f96700-2d4a-4b30-ada0-04cf70b7f6d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n\" (UID: \"c2f96700-2d4a-4b30-ada0-04cf70b7f6d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n"
Oct 29 22:07:05.260315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.224433    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f96700-2d4a-4b30-ada0-04cf70b7f6d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n\" (UID: \"c2f96700-2d4a-4b30-ada0-04cf70b7f6d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n"
Oct 29 22:07:05.260315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.255769    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45b4cce5-7740-4811-9191-9bfddb5e3fac-plugins-config-volume" pod "45b4cce5-7740-4811-9191-9bfddb5e3fac" (UID: "45b4cce5-7740-4811-9191-9bfddb5e3fac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:07:05.276796 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.276525    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.315516 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.294836    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.339347 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.300714    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b4cce5-7740-4811-9191-9bfddb5e3fac-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:07:05.459912 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.456442    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj" event={"ID":"45b4cce5-7740-4811-9191-9bfddb5e3fac","Type":"ContainerDied","Data":"65b1e218856dc9f6c94f97714d3d504245f927341e99ff19d0688da84e6b3227"}
Oct 29 22:07:05.460784 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.460645    4962 scope.go:122] "RemoveContainer" containerID="9ed2aac9a09967bf8a8e9d07727c9efb6d18b181612d36ad7f0560308fcde426"
Oct 29 22:07:05.565313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.561888    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.619430 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.574410    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.619430 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.576203    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.725657 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.623362    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.791311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.789817    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:07:05.791311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.789879    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5" podUID="a437f4e2-dc10-41de-851a-ee488b08c6f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.825688 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.793270    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.825688 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.793411    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.825688 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.794876    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.827751 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.793281    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.890629 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.878490    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:05.890629 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.879138    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw"]
Oct 29 22:07:05.890629 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.879185    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f6171d-d6588cdfb-9pcbw"]
Oct 29 22:07:05.890629 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.879312    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c"]
Oct 29 22:07:05.890629 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.879336    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f6171d-58b4df49f9-7r69c"]
Oct 29 22:07:05.933348 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.911579    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8lrn\" (UniqueName: \"kubernetes.io/projected/c2f96700-2d4a-4b30-ada0-04cf70b7f6d2-kube-api-access-x8lrn\") pod \"payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n\" (UID: \"c2f96700-2d4a-4b30-ada0-04cf70b7f6d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n"
Oct 29 22:07:05.933348 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.915510    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b4cce5-7740-4811-9191-9bfddb5e3fac-kube-api-access-rb8m2" pod "45b4cce5-7740-4811-9191-9bfddb5e3fac" (UID: "45b4cce5-7740-4811-9191-9bfddb5e3fac"). InnerVolumeSpecName "kube-api-access-rb8m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:07:05.951954 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:05.951915    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb8m2\" (UniqueName: \"kubernetes.io/projected/45b4cce5-7740-4811-9191-9bfddb5e3fac-kube-api-access-rb8m2\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:07:06.211568 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:06.155872    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 22:07:06.211568 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:06.202020    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n"
Oct 29 22:07:06.304029 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:06.303458    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:07:06.314705 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:06.314676    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:07:06.439417 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:06.439381    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj"
Oct 29 22:07:06.845597 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:06.585256    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:06.845597 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:06.593316    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:07:06.845597 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:06.638431    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:06.845597 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:06.734235    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:06.845597 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:06.831785    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:06.845597 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:06.832403    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:07:06.845597 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:06.837347    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:07.021896 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:07.020809    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 22:07:07.283973 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:07:07.276791    4962 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b4cce5_7740_4811_9191_9bfddb5e3fac.slice/crio-65b1e218856dc9f6c94f97714d3d504245f927341e99ff19d0688da84e6b3227\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b4cce5_7740_4811_9191_9bfddb5e3fac.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 22:07:07.304570 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:07.304537    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="263a0c66-6465-4269-b15c-28ccb8afbe32" path="/var/lib/kubelet/pods/263a0c66-6465-4269-b15c-28ccb8afbe32/volumes"
Oct 29 22:07:07.305701 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:07.305677    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d397882-c29a-47e6-a6d5-280f529d89d6" path="/var/lib/kubelet/pods/2d397882-c29a-47e6-a6d5-280f529d89d6/volumes"
Oct 29 22:07:07.310996 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:07:07.307991    4962 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b4cce5_7740_4811_9191_9bfddb5e3fac.slice/crio-65b1e218856dc9f6c94f97714d3d504245f927341e99ff19d0688da84e6b3227\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b4cce5_7740_4811_9191_9bfddb5e3fac.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 22:07:07.326367 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:07.319198    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 22:07:07.326843 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:07.326820    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:07:07.327647 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:07.327628    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:07:07.327846 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:07.327834    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:07:07.329768 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:07.329753    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:07:07.334231 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:07.331884    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 22:07:07.343577 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:07.342725    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 22:07:07.354273 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:07.354243    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj"]
Oct 29 22:07:07.371632 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:07.371565    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d53cd-c7f6655f5-8bvbj"]
Oct 29 22:07:07.433857 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:07:07.433826    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6fe16e8_3928_4766_9daf_7a449c61233d.slice/crio-739c5ac37be0f04cfd8ae45f69ccd695b7b57565844dbb01855090fcd69890c9 WatchSource:0}: Error finding container 739c5ac37be0f04cfd8ae45f69ccd695b7b57565844dbb01855090fcd69890c9: Status 404 returned error can't find the container with id 739c5ac37be0f04cfd8ae45f69ccd695b7b57565844dbb01855090fcd69890c9
Oct 29 22:07:07.440658 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:07.440633    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847"]
Oct 29 22:07:07.483366 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:07.483322    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847" event={"ID":"e6fe16e8-3928-4766-9daf-7a449c61233d","Type":"ContainerStarted","Data":"739c5ac37be0f04cfd8ae45f69ccd695b7b57565844dbb01855090fcd69890c9"}
Oct 29 22:07:07.578781 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:07:07.578729    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f96700_2d4a_4b30_ada0_04cf70b7f6d2.slice/crio-e98c4fada6bebd9b4d42317d0d436b222371ff6d4cf11492f18e54825143b54b WatchSource:0}: Error finding container e98c4fada6bebd9b4d42317d0d436b222371ff6d4cf11492f18e54825143b54b: Status 404 returned error can't find the container with id e98c4fada6bebd9b4d42317d0d436b222371ff6d4cf11492f18e54825143b54b
Oct 29 22:07:07.584775 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:07.584751    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n"]
Oct 29 22:07:08.493723 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:08.493686    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n" event={"ID":"c2f96700-2d4a-4b30-ada0-04cf70b7f6d2","Type":"ContainerStarted","Data":"e8b9f21452e1283dc320b501e5470c07c9a9247e2d2b7535392d8912fd9a38e9"}
Oct 29 22:07:08.494204 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:08.493761    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n"
Oct 29 22:07:08.494204 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:08.493788    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n" event={"ID":"c2f96700-2d4a-4b30-ada0-04cf70b7f6d2","Type":"ContainerStarted","Data":"e98c4fada6bebd9b4d42317d0d436b222371ff6d4cf11492f18e54825143b54b"}
Oct 29 22:07:08.495659 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:08.495629    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847" event={"ID":"e6fe16e8-3928-4766-9daf-7a449c61233d","Type":"ContainerStarted","Data":"6ee7b4fb3db237dea450cb090ad407e84a85183c300362d3b8070fed57aac232"}
Oct 29 22:07:08.495792 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:08.495732    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847"
Oct 29 22:07:08.543362 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:08.543310    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n" podStartSLOduration=7.5432820419999995 podStartE2EDuration="7.543282042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:08.542349638 +0000 UTC m=+1249.615306626" watchObservedRunningTime="2026-10-29 22:07:08.543282042 +0000 UTC m=+1249.616239005"
Oct 29 22:07:08.591196 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:08.591148    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847" podStartSLOduration=7.5911356229999996 podStartE2EDuration="7.591135623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:08.591095695 +0000 UTC m=+1249.664052659" watchObservedRunningTime="2026-10-29 22:07:08.591135623 +0000 UTC m=+1249.664092586"
Oct 29 22:07:09.060750 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:09.060719    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45b4cce5-7740-4811-9191-9bfddb5e3fac" path="/var/lib/kubelet/pods/45b4cce5-7740-4811-9191-9bfddb5e3fac/volumes"
Oct 29 22:07:09.727604 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:07:09.727569    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:07:10.334103 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:10.334069    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9"]
Oct 29 22:07:10.334949 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:10.334929    4962 cpu_manager.go:414] "containerMap: removing container" podUID="45b4cce5-7740-4811-9191-9bfddb5e3fac" containerName="payload-pre-processing"
Oct 29 22:07:10.334949 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:10.334950    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45b4cce5-7740-4811-9191-9bfddb5e3fac" containerName="payload-pre-processing"
Oct 29 22:07:10.335116 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:10.335107    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="45b4cce5-7740-4811-9191-9bfddb5e3fac" containerName="payload-pre-processing"
Oct 29 22:07:10.339223 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:10.339195    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9"
Oct 29 22:07:10.362062 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:10.362038    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9"]
Oct 29 22:07:10.367399 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:10.367377    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-df42c7\""
Oct 29 22:07:10.390741 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:10.390717    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn"]
Oct 29 22:07:10.393482 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:10.393439    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd2c6fb-be8c-48dd-9c74-e71293a09672-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9\" (UID: \"6dd2c6fb-be8c-48dd-9c74-e71293a09672\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9"
Oct 29 22:07:10.393774 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:10.393732    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqd58\" (UniqueName: \"kubernetes.io/projected/6dd2c6fb-be8c-48dd-9c74-e71293a09672-kube-api-access-rqd58\") pod \"payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9\" (UID: \"6dd2c6fb-be8c-48dd-9c74-e71293a09672\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9"
Oct 29 22:07:10.396068 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:10.396052    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn"
Oct 29 22:07:10.422798 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:10.422777    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn"]
Oct 29 22:07:10.426373 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:10.426355    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-df42c7-dockercfg-qbv88\""
Oct 29 22:07:10.550337 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:10.509495    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94fff\" (UniqueName: \"kubernetes.io/projected/28f4dbbf-6c33-49b8-a263-bf5956d62247-kube-api-access-94fff\") pod \"payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn\" (UID: \"28f4dbbf-6c33-49b8-a263-bf5956d62247\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn"
Oct 29 22:07:10.564483 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:10.556270    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd2c6fb-be8c-48dd-9c74-e71293a09672-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9\" (UID: \"6dd2c6fb-be8c-48dd-9c74-e71293a09672\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9"
Oct 29 22:07:10.564483 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:10.556473    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28f4dbbf-6c33-49b8-a263-bf5956d62247-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn\" (UID: \"28f4dbbf-6c33-49b8-a263-bf5956d62247\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn"
Oct 29 22:07:10.564483 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:10.556523    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqd58\" (UniqueName: \"kubernetes.io/projected/6dd2c6fb-be8c-48dd-9c74-e71293a09672-kube-api-access-rqd58\") pod \"payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9\" (UID: \"6dd2c6fb-be8c-48dd-9c74-e71293a09672\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9"
Oct 29 22:07:10.564483 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:10.558169    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd2c6fb-be8c-48dd-9c74-e71293a09672-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9\" (UID: \"6dd2c6fb-be8c-48dd-9c74-e71293a09672\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9"
Oct 29 22:07:10.630354 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:10.627421    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqd58\" (UniqueName: \"kubernetes.io/projected/6dd2c6fb-be8c-48dd-9c74-e71293a09672-kube-api-access-rqd58\") pod \"payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9\" (UID: \"6dd2c6fb-be8c-48dd-9c74-e71293a09672\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9"
Oct 29 22:07:10.697937 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:10.680339    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94fff\" (UniqueName: \"kubernetes.io/projected/28f4dbbf-6c33-49b8-a263-bf5956d62247-kube-api-access-94fff\") pod \"payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn\" (UID: \"28f4dbbf-6c33-49b8-a263-bf5956d62247\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn"
Oct 29 22:07:10.697937 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:10.680525    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28f4dbbf-6c33-49b8-a263-bf5956d62247-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn\" (UID: \"28f4dbbf-6c33-49b8-a263-bf5956d62247\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn"
Oct 29 22:07:10.697937 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:10.681589    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28f4dbbf-6c33-49b8-a263-bf5956d62247-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn\" (UID: \"28f4dbbf-6c33-49b8-a263-bf5956d62247\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn"
Oct 29 22:07:10.697937 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:10.696700    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9"
Oct 29 22:07:10.747309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:10.743443    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94fff\" (UniqueName: \"kubernetes.io/projected/28f4dbbf-6c33-49b8-a263-bf5956d62247-kube-api-access-94fff\") pod \"payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn\" (UID: \"28f4dbbf-6c33-49b8-a263-bf5956d62247\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn"
Oct 29 22:07:11.011192 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.011165    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn"
Oct 29 22:07:11.268323 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:07:11.268259    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd2c6fb_be8c_48dd_9c74_e71293a09672.slice/crio-09327c8df7373b4418967f6848b77031179dc0962b6322713cc41d0b803df10f WatchSource:0}: Error finding container 09327c8df7373b4418967f6848b77031179dc0962b6322713cc41d0b803df10f: Status 404 returned error can't find the container with id 09327c8df7373b4418967f6848b77031179dc0962b6322713cc41d0b803df10f
Oct 29 22:07:11.268574 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.268548    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9"]
Oct 29 22:07:11.374172 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:07:11.374144    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f4dbbf_6c33_49b8_a263_bf5956d62247.slice/crio-03e5abeea91903834f90b2d713d6b5f35ebfe26e603628f581e85d57ede754cc WatchSource:0}: Error finding container 03e5abeea91903834f90b2d713d6b5f35ebfe26e603628f581e85d57ede754cc: Status 404 returned error can't find the container with id 03e5abeea91903834f90b2d713d6b5f35ebfe26e603628f581e85d57ede754cc
Oct 29 22:07:11.374989 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.374861    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn"]
Oct 29 22:07:11.470326 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.470268    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k"]
Oct 29 22:07:11.470849 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.470456    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k" podUID="3c7978e3-558b-4148-b0fc-3e275473a211" containerName="maas-api" containerID="cri-o://91693487f5669a230f61e5d972d8dff350fdb629c9d55173532e7cefc11e2e14" gracePeriod=30
Oct 29 22:07:11.639651 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.639616    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k" podUID="3c7978e3-558b-4148-b0fc-3e275473a211" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.106:8443/health\": dial tcp 10.132.0.106:8443: connect: connection refused"
Oct 29 22:07:11.647062 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.647003    4962 generic.go:356] "Generic (PLEG): container finished" podID="3c7978e3-558b-4148-b0fc-3e275473a211" containerID="91693487f5669a230f61e5d972d8dff350fdb629c9d55173532e7cefc11e2e14" exitCode=0
Oct 29 22:07:11.647346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.647319    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k" event={"ID":"3c7978e3-558b-4148-b0fc-3e275473a211","Type":"ContainerDied","Data":"91693487f5669a230f61e5d972d8dff350fdb629c9d55173532e7cefc11e2e14"}
Oct 29 22:07:11.648942 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.648918    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9" event={"ID":"6dd2c6fb-be8c-48dd-9c74-e71293a09672","Type":"ContainerStarted","Data":"ed59f936975e2aecc54510e1ac9ccdb972139caf722be54fd6787258923e05ae"}
Oct 29 22:07:11.649037 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.648964    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9" event={"ID":"6dd2c6fb-be8c-48dd-9c74-e71293a09672","Type":"ContainerStarted","Data":"09327c8df7373b4418967f6848b77031179dc0962b6322713cc41d0b803df10f"}
Oct 29 22:07:11.649037 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.649023    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9"
Oct 29 22:07:11.650735 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.650712    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn" event={"ID":"28f4dbbf-6c33-49b8-a263-bf5956d62247","Type":"ContainerStarted","Data":"aacc559172b4ba3274ec9e34f1497e801ca93cbd1c7dfd8c1d9d68f681c6fdf3"}
Oct 29 22:07:11.650840 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.650749    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn" event={"ID":"28f4dbbf-6c33-49b8-a263-bf5956d62247","Type":"ContainerStarted","Data":"03e5abeea91903834f90b2d713d6b5f35ebfe26e603628f581e85d57ede754cc"}
Oct 29 22:07:11.650840 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.650779    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn"
Oct 29 22:07:11.749518 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.749470    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9" podStartSLOduration=1.749456506 podStartE2EDuration="1.749456506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:11.698522136 +0000 UTC m=+1252.771479101" watchObservedRunningTime="2026-10-29 22:07:11.749456506 +0000 UTC m=+1252.822413470"
Oct 29 22:07:11.749976 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.749547    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn" podStartSLOduration=1.749542699 podStartE2EDuration="1.749542699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:11.745980946 +0000 UTC m=+1252.818937911" watchObservedRunningTime="2026-10-29 22:07:11.749542699 +0000 UTC m=+1252.822499663"
Oct 29 22:07:11.776765 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.776743    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k"
Oct 29 22:07:11.804738 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.804708    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c7978e3-558b-4148-b0fc-3e275473a211-kube-api-access-krmvt\" (UniqueName: \"kubernetes.io/projected/3c7978e3-558b-4148-b0fc-3e275473a211-kube-api-access-krmvt\") pod \"3c7978e3-558b-4148-b0fc-3e275473a211\" (UID: \"3c7978e3-558b-4148-b0fc-3e275473a211\") "
Oct 29 22:07:11.804841 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.804794    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c7978e3-558b-4148-b0fc-3e275473a211-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c7978e3-558b-4148-b0fc-3e275473a211-maas-api-tls\") pod \"3c7978e3-558b-4148-b0fc-3e275473a211\" (UID: \"3c7978e3-558b-4148-b0fc-3e275473a211\") "
Oct 29 22:07:11.804943 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.804929    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c7978e3-558b-4148-b0fc-3e275473a211-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c7978e3-558b-4148-b0fc-3e275473a211-metrics-certs\") pod \"3c7978e3-558b-4148-b0fc-3e275473a211\" (UID: \"3c7978e3-558b-4148-b0fc-3e275473a211\") "
Oct 29 22:07:11.805092 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.805064    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c7978e3-558b-4148-b0fc-3e275473a211-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7978e3-558b-4148-b0fc-3e275473a211-openshift-service-ca\") pod \"3c7978e3-558b-4148-b0fc-3e275473a211\" (UID: \"3c7978e3-558b-4148-b0fc-3e275473a211\") "
Oct 29 22:07:11.805486 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.805461    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c7978e3-558b-4148-b0fc-3e275473a211-openshift-service-ca" pod "3c7978e3-558b-4148-b0fc-3e275473a211" (UID: "3c7978e3-558b-4148-b0fc-3e275473a211"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:07:11.805850 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.805829    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c7978e3-558b-4148-b0fc-3e275473a211-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:07:11.806764 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.806739    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c7978e3-558b-4148-b0fc-3e275473a211-kube-api-access-krmvt" pod "3c7978e3-558b-4148-b0fc-3e275473a211" (UID: "3c7978e3-558b-4148-b0fc-3e275473a211"). InnerVolumeSpecName "kube-api-access-krmvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:07:11.807031 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.807000    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c7978e3-558b-4148-b0fc-3e275473a211-metrics-certs" pod "3c7978e3-558b-4148-b0fc-3e275473a211" (UID: "3c7978e3-558b-4148-b0fc-3e275473a211"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:07:11.807126 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.807028    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c7978e3-558b-4148-b0fc-3e275473a211-maas-api-tls" pod "3c7978e3-558b-4148-b0fc-3e275473a211" (UID: "3c7978e3-558b-4148-b0fc-3e275473a211"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:07:11.907154 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.907125    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krmvt\" (UniqueName: \"kubernetes.io/projected/3c7978e3-558b-4148-b0fc-3e275473a211-kube-api-access-krmvt\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:07:11.907154 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.907153    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c7978e3-558b-4148-b0fc-3e275473a211-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:07:11.907418 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:11.907163    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c7978e3-558b-4148-b0fc-3e275473a211-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:07:12.388364 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.387539    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"]
Oct 29 22:07:12.408566 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.405882    4962 cpu_manager.go:414] "containerMap: removing container" podUID="3c7978e3-558b-4148-b0fc-3e275473a211" containerName="maas-api"
Oct 29 22:07:12.408566 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.405915    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c7978e3-558b-4148-b0fc-3e275473a211" containerName="maas-api"
Oct 29 22:07:12.408566 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.406155    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c7978e3-558b-4148-b0fc-3e275473a211" containerName="maas-api"
Oct 29 22:07:12.429841 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.427032    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.432599 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.431850    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"]
Oct 29 22:07:12.470107 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.470074    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-workload-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.470251 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.470120    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-podinfo\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.470251 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.470158    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-envoy\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.470467 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.470440    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t77nk\" (UniqueName: \"kubernetes.io/projected/8f0235b0-8935-415c-a47e-dd3ac5df7a84-kube-api-access-t77nk\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.470568 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.470547    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-credential-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.470704 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.470686    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-token\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.470830 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.470808    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-data\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.470922 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.470880    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-workload-certs\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.470978 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.470943    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istiod-ca-cert\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.575314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.572356    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-token\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.575314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.572440    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-data\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.575314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.572511    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-workload-certs\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.575314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.572574    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istiod-ca-cert\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.575314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.572654    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-workload-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.575314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.572681    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-podinfo\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.575314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.572713    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-envoy\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.575314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.572765    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t77nk\" (UniqueName: \"kubernetes.io/projected/8f0235b0-8935-415c-a47e-dd3ac5df7a84-kube-api-access-t77nk\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.575314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.572858    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-credential-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.575314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.574130    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istiod-ca-cert\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.575314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.574785    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-data\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.575314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.575016    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-workload-certs\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.581960 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.576708    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-workload-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.581960 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.577627    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-credential-socket\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.586355 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.582701    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-podinfo\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.624216 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.601605    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-envoy\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.624216 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.623527    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-token\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.656625 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.656021    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t77nk\" (UniqueName: \"kubernetes.io/projected/8f0235b0-8935-415c-a47e-dd3ac5df7a84-kube-api-access-t77nk\") pod \"e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") " pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:12.687542 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.687503    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k" event={"ID":"3c7978e3-558b-4148-b0fc-3e275473a211","Type":"ContainerDied","Data":"ef5a34c6f34edc0c308dc83bee2d549acec9515792f74374d486c1c5532c9e4d"}
Oct 29 22:07:12.687715 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.687576    4962 scope.go:122] "RemoveContainer" containerID="91693487f5669a230f61e5d972d8dff350fdb629c9d55173532e7cefc11e2e14"
Oct 29 22:07:12.687715 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.687620    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k"
Oct 29 22:07:12.743407 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.743383    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k"]
Oct 29 22:07:12.748531 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.748499    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-751a7416-56467cbd84-k474k"]
Oct 29 22:07:12.748723 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:12.748705    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:13.061364 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:13.061273    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c7978e3-558b-4148-b0fc-3e275473a211" path="/var/lib/kubelet/pods/3c7978e3-558b-4148-b0fc-3e275473a211/volumes"
Oct 29 22:07:13.083409 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:07:13.083375    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0235b0_8935_415c_a47e_dd3ac5df7a84.slice/crio-94e1afb28b82395174586d2bad5b08052562e5cf29ee8b20a6e2441382684598 WatchSource:0}: Error finding container 94e1afb28b82395174586d2bad5b08052562e5cf29ee8b20a6e2441382684598: Status 404 returned error can't find the container with id 94e1afb28b82395174586d2bad5b08052562e5cf29ee8b20a6e2441382684598
Oct 29 22:07:13.084089 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:13.084000    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"]
Oct 29 22:07:13.086070 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:13.086037    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:07:13.086148 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:13.086103    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:07:13.086148 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:13.086134    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:07:13.700095 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:13.700060    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4" event={"ID":"8f0235b0-8935-415c-a47e-dd3ac5df7a84","Type":"ContainerStarted","Data":"43fe6a5d691049d163e10e716ebacc44ffe4835dcee189078bf88610948b9dce"}
Oct 29 22:07:13.700275 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:13.700146    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4" event={"ID":"8f0235b0-8935-415c-a47e-dd3ac5df7a84","Type":"ContainerStarted","Data":"94e1afb28b82395174586d2bad5b08052562e5cf29ee8b20a6e2441382684598"}
Oct 29 22:07:13.748777 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:13.748751    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:13.764527 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:13.764479    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4" podStartSLOduration=1.764466198 podStartE2EDuration="1.764466198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:13.76081592 +0000 UTC m=+1254.833772883" watchObservedRunningTime="2026-10-29 22:07:13.764466198 +0000 UTC m=+1254.837423162"
Oct 29 22:07:14.750663 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:14.750625    4962 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 29 22:07:14.751119 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:14.750682    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4" podUID="8f0235b0-8935-415c-a47e-dd3ac5df7a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 29 22:07:15.749657 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:15.749622    4962 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 29 22:07:15.749854 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:15.749733    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4" podUID="8f0235b0-8935-415c-a47e-dd3ac5df7a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 29 22:07:16.017646 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:16.017556    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9"]
Oct 29 22:07:16.055058 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:16.055030    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4"]
Oct 29 22:07:16.770311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:16.766748    4962 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 29 22:07:16.770311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:16.767008    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4" podUID="8f0235b0-8935-415c-a47e-dd3ac5df7a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 29 22:07:17.749852 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:17.749819    4962 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 29 22:07:17.750210 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:17.749878    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4" podUID="8f0235b0-8935-415c-a47e-dd3ac5df7a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 29 22:07:18.749875 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:18.749839    4962 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 29 22:07:18.750383 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:18.749888    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4" podUID="8f0235b0-8935-415c-a47e-dd3ac5df7a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 29 22:07:19.509700 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:19.509670    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n"
Oct 29 22:07:19.510081 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:19.510060    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847"
Oct 29 22:07:19.749734 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:19.749699    4962 patch_prober.go:29] interesting pod/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 29 22:07:19.749983 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:19.749955    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4" podUID="8f0235b0-8935-415c-a47e-dd3ac5df7a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 29 22:07:20.754981 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:20.754948    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:20.755458 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:20.755440    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:20.756095 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:20.756074    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:07:21.018438 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.018349    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9" podUID="f30ae4be-1795-4728-b8b6-075038cff8ff" containerName="payload-processing" containerID="cri-o://182616628e67876ba3eaf7179da98bf2461cd41341516f1c3e09b188dac4c589" gracePeriod=25
Oct 29 22:07:21.055255 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.055222    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4" podUID="37af084a-d949-4a30-a867-54e3cc3943b9" containerName="payload-pre-processing" containerID="cri-o://2fc7c496b4ddcb57dba46a3ec0cdbd3ce4182eb095ac0fad37b3ca47e4fce76d" gracePeriod=25
Oct 29 22:07:21.392989 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.392963    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9"
Oct 29 22:07:21.396618 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.396597    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4"
Oct 29 22:07:21.658314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.655025    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f30ae4be-1795-4728-b8b6-075038cff8ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30ae4be-1795-4728-b8b6-075038cff8ff-plugins-config-volume\") pod \"f30ae4be-1795-4728-b8b6-075038cff8ff\" (UID: \"f30ae4be-1795-4728-b8b6-075038cff8ff\") "
Oct 29 22:07:21.658314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.655142    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37af084a-d949-4a30-a867-54e3cc3943b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37af084a-d949-4a30-a867-54e3cc3943b9-plugins-config-volume\") pod \"37af084a-d949-4a30-a867-54e3cc3943b9\" (UID: \"37af084a-d949-4a30-a867-54e3cc3943b9\") "
Oct 29 22:07:21.658314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.655481    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37af084a-d949-4a30-a867-54e3cc3943b9-kube-api-access-5sw7m\" (UniqueName: \"kubernetes.io/projected/37af084a-d949-4a30-a867-54e3cc3943b9-kube-api-access-5sw7m\") pod \"37af084a-d949-4a30-a867-54e3cc3943b9\" (UID: \"37af084a-d949-4a30-a867-54e3cc3943b9\") "
Oct 29 22:07:21.658314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.655601    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f30ae4be-1795-4728-b8b6-075038cff8ff-kube-api-access-rmgmq\" (UniqueName: \"kubernetes.io/projected/f30ae4be-1795-4728-b8b6-075038cff8ff-kube-api-access-rmgmq\") pod \"f30ae4be-1795-4728-b8b6-075038cff8ff\" (UID: \"f30ae4be-1795-4728-b8b6-075038cff8ff\") "
Oct 29 22:07:21.681048 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.670117    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30ae4be-1795-4728-b8b6-075038cff8ff-plugins-config-volume" pod "f30ae4be-1795-4728-b8b6-075038cff8ff" (UID: "f30ae4be-1795-4728-b8b6-075038cff8ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:07:21.681048 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.671117    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37af084a-d949-4a30-a867-54e3cc3943b9-plugins-config-volume" pod "37af084a-d949-4a30-a867-54e3cc3943b9" (UID: "37af084a-d949-4a30-a867-54e3cc3943b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:07:21.681048 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.671483    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f30ae4be-1795-4728-b8b6-075038cff8ff-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:07:21.681048 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.671506    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37af084a-d949-4a30-a867-54e3cc3943b9-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:07:21.682783 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.682739    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30ae4be-1795-4728-b8b6-075038cff8ff-kube-api-access-rmgmq" pod "f30ae4be-1795-4728-b8b6-075038cff8ff" (UID: "f30ae4be-1795-4728-b8b6-075038cff8ff"). InnerVolumeSpecName "kube-api-access-rmgmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:07:21.818939 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.804275    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmgmq\" (UniqueName: \"kubernetes.io/projected/f30ae4be-1795-4728-b8b6-075038cff8ff-kube-api-access-rmgmq\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:07:21.832168 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.832117    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37af084a-d949-4a30-a867-54e3cc3943b9-kube-api-access-5sw7m" pod "37af084a-d949-4a30-a867-54e3cc3943b9" (UID: "37af084a-d949-4a30-a867-54e3cc3943b9"). InnerVolumeSpecName "kube-api-access-5sw7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:07:21.895652 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.895614    4962 generic.go:356] "Generic (PLEG): container finished" podID="37af084a-d949-4a30-a867-54e3cc3943b9" containerID="2fc7c496b4ddcb57dba46a3ec0cdbd3ce4182eb095ac0fad37b3ca47e4fce76d" exitCode=0
Oct 29 22:07:21.895801 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.895663    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4"
Oct 29 22:07:21.895801 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.895686    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4" event={"ID":"37af084a-d949-4a30-a867-54e3cc3943b9","Type":"ContainerDied","Data":"2fc7c496b4ddcb57dba46a3ec0cdbd3ce4182eb095ac0fad37b3ca47e4fce76d"}
Oct 29 22:07:21.895801 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.895739    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4" event={"ID":"37af084a-d949-4a30-a867-54e3cc3943b9","Type":"ContainerDied","Data":"412a87c279d638f5f6105ec2f521f1f5af809d13458d357db4c32e509404fb00"}
Oct 29 22:07:21.895801 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.895760    4962 scope.go:122] "RemoveContainer" containerID="2fc7c496b4ddcb57dba46a3ec0cdbd3ce4182eb095ac0fad37b3ca47e4fce76d"
Oct 29 22:07:21.897687 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.897651    4962 generic.go:356] "Generic (PLEG): container finished" podID="f30ae4be-1795-4728-b8b6-075038cff8ff" containerID="182616628e67876ba3eaf7179da98bf2461cd41341516f1c3e09b188dac4c589" exitCode=0
Oct 29 22:07:21.897687 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.897675    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9" event={"ID":"f30ae4be-1795-4728-b8b6-075038cff8ff","Type":"ContainerDied","Data":"182616628e67876ba3eaf7179da98bf2461cd41341516f1c3e09b188dac4c589"}
Oct 29 22:07:21.897834 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.897699    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9"
Oct 29 22:07:21.897834 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.897724    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9" event={"ID":"f30ae4be-1795-4728-b8b6-075038cff8ff","Type":"ContainerDied","Data":"acbd2e379e332db25e4d74ac913377cc4d4da2974112059726e4ec2a3804a0e0"}
Oct 29 22:07:21.907344 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.907298    4962 scope.go:122] "RemoveContainer" containerID="2fc7c496b4ddcb57dba46a3ec0cdbd3ce4182eb095ac0fad37b3ca47e4fce76d"
Oct 29 22:07:21.907635 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:07:21.907613    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc7c496b4ddcb57dba46a3ec0cdbd3ce4182eb095ac0fad37b3ca47e4fce76d\": container with ID starting with 2fc7c496b4ddcb57dba46a3ec0cdbd3ce4182eb095ac0fad37b3ca47e4fce76d not found: ID does not exist" containerID="2fc7c496b4ddcb57dba46a3ec0cdbd3ce4182eb095ac0fad37b3ca47e4fce76d"
Oct 29 22:07:21.907733 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.907643    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc7c496b4ddcb57dba46a3ec0cdbd3ce4182eb095ac0fad37b3ca47e4fce76d"} err="failed to get container status \"2fc7c496b4ddcb57dba46a3ec0cdbd3ce4182eb095ac0fad37b3ca47e4fce76d\": rpc error: code = NotFound desc = could not find container \"2fc7c496b4ddcb57dba46a3ec0cdbd3ce4182eb095ac0fad37b3ca47e4fce76d\": container with ID starting with 2fc7c496b4ddcb57dba46a3ec0cdbd3ce4182eb095ac0fad37b3ca47e4fce76d not found: ID does not exist"
Oct 29 22:07:21.907733 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.907659    4962 scope.go:122] "RemoveContainer" containerID="182616628e67876ba3eaf7179da98bf2461cd41341516f1c3e09b188dac4c589"
Oct 29 22:07:21.919423 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.919403    4962 scope.go:122] "RemoveContainer" containerID="182616628e67876ba3eaf7179da98bf2461cd41341516f1c3e09b188dac4c589"
Oct 29 22:07:21.919761 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:07:21.919723    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"182616628e67876ba3eaf7179da98bf2461cd41341516f1c3e09b188dac4c589\": container with ID starting with 182616628e67876ba3eaf7179da98bf2461cd41341516f1c3e09b188dac4c589 not found: ID does not exist" containerID="182616628e67876ba3eaf7179da98bf2461cd41341516f1c3e09b188dac4c589"
Oct 29 22:07:21.919850 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.919753    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"182616628e67876ba3eaf7179da98bf2461cd41341516f1c3e09b188dac4c589"} err="failed to get container status \"182616628e67876ba3eaf7179da98bf2461cd41341516f1c3e09b188dac4c589\": rpc error: code = NotFound desc = could not find container \"182616628e67876ba3eaf7179da98bf2461cd41341516f1c3e09b188dac4c589\": container with ID starting with 182616628e67876ba3eaf7179da98bf2461cd41341516f1c3e09b188dac4c589 not found: ID does not exist"
Oct 29 22:07:21.920377 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.920355    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sw7m\" (UniqueName: \"kubernetes.io/projected/37af084a-d949-4a30-a867-54e3cc3943b9-kube-api-access-5sw7m\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:07:21.958326 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.958297    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9"]
Oct 29 22:07:21.969902 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:21.969875    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-380eea-79d98c865d-9tqd9"]
Oct 29 22:07:22.021410 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:22.021389    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4"]
Oct 29 22:07:22.025718 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:22.025696    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-380eea-559cdd6f7f-pmhz4"]
Oct 29 22:07:22.691418 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:22.691390    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9"
Oct 29 22:07:22.691586 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:22.691470    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn"
Oct 29 22:07:23.060342 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:23.060264    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37af084a-d949-4a30-a867-54e3cc3943b9" path="/var/lib/kubelet/pods/37af084a-d949-4a30-a867-54e3cc3943b9/volumes"
Oct 29 22:07:23.060745 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:23.060730    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f30ae4be-1795-4728-b8b6-075038cff8ff" path="/var/lib/kubelet/pods/f30ae4be-1795-4728-b8b6-075038cff8ff/volumes"
Oct 29 22:07:28.502222 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.502186    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"]
Oct 29 22:07:28.503082 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.503062    4962 cpu_manager.go:414] "containerMap: removing container" podUID="f30ae4be-1795-4728-b8b6-075038cff8ff" containerName="payload-processing"
Oct 29 22:07:28.503082 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.503082    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f30ae4be-1795-4728-b8b6-075038cff8ff" containerName="payload-processing"
Oct 29 22:07:28.503193 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.503098    4962 cpu_manager.go:414] "containerMap: removing container" podUID="37af084a-d949-4a30-a867-54e3cc3943b9" containerName="payload-pre-processing"
Oct 29 22:07:28.503193 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.503103    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37af084a-d949-4a30-a867-54e3cc3943b9" containerName="payload-pre-processing"
Oct 29 22:07:28.503305 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.503235    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="f30ae4be-1795-4728-b8b6-075038cff8ff" containerName="payload-processing"
Oct 29 22:07:28.503305 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.503243    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="37af084a-d949-4a30-a867-54e3cc3943b9" containerName="payload-pre-processing"
Oct 29 22:07:28.508654 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.508633    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.533557 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.533530    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"]
Oct 29 22:07:28.538584 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.538565    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.540016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.539993    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-380eea-openshift-default-dockercfg-lm8cf\""
Oct 29 22:07:28.540931 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.540904    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-token\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.541112 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.541090    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-workload-certs\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.541206 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.541176    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istiod-ca-cert\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.541255 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.541202    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsgg2\" (UniqueName: \"kubernetes.io/projected/9b3220a5-a632-40bd-b9ad-668bb2b5b237-kube-api-access-bsgg2\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.541343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.541325    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-workload-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.541529 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.541505    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-credential-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.541670 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.541652    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-envoy\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.541737 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.541693    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-podinfo\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.541849 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.541826    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-data\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.552763 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.552740    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"]
Oct 29 22:07:28.579594 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.578981    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"]
Oct 29 22:07:28.643756 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.643726    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1106f45f-cdca-428f-8a2e-dc402d199ae8-istiod-ca-cert\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.643907 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.643765    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-workload-certs\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.643907 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.643851    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-credential-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.644023 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.643913    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istiod-ca-cert\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.644023 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.643977    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsgg2\" (UniqueName: \"kubernetes.io/projected/9b3220a5-a632-40bd-b9ad-668bb2b5b237-kube-api-access-bsgg2\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.644146 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.644113    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-workload-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.644199 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.644162    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zzdx\" (UniqueName: \"kubernetes.io/projected/1106f45f-cdca-428f-8a2e-dc402d199ae8-kube-api-access-2zzdx\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.644255 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.644161    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-workload-certs\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.644342 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.644248    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-credential-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.644408 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.644351    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-podinfo\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.644408 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.644396    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-envoy\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.644520 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.644424    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-workload-certs\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.644520 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.644452    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-workload-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.644520 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.644500    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-data\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.644683 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.644544    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-envoy\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.644683 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.644571    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-podinfo\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.644683 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.644600    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-credential-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.644683 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.644655    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-data\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.644880 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.644703    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-token\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.644880 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.644753    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istiod-ca-cert\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.644880 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.644791    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-workload-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.644880 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.644827    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-token\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.645059 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.644960    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-data\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.646868 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.646844    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-envoy\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.647502 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.647482    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-podinfo\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.680554 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.680530    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-token\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.687081 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.687054    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsgg2\" (UniqueName: \"kubernetes.io/projected/9b3220a5-a632-40bd-b9ad-668bb2b5b237-kube-api-access-bsgg2\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.746178 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.746143    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-token\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.746353 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.746211    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-workload-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.746353 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.746340    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1106f45f-cdca-428f-8a2e-dc402d199ae8-istiod-ca-cert\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.746453 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.746388    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-credential-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.746545 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.746526    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zzdx\" (UniqueName: \"kubernetes.io/projected/1106f45f-cdca-428f-8a2e-dc402d199ae8-kube-api-access-2zzdx\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.746632 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.746616    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-podinfo\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.746696 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.746658    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-envoy\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.746696 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.746686    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-workload-certs\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.746798 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.746702    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-workload-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.746798 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.746724    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-data\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.746798 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.746755    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-credential-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.747012 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.746992    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-workload-certs\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.747104 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.747088    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-data\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.747157 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.747105    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1106f45f-cdca-428f-8a2e-dc402d199ae8-istiod-ca-cert\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.748912 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.748884    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-envoy\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.749167 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.749147    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-podinfo\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.771708 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.771659    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zzdx\" (UniqueName: \"kubernetes.io/projected/1106f45f-cdca-428f-8a2e-dc402d199ae8-kube-api-access-2zzdx\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.771986 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.771970    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-token\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:28.822929 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.822904    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:28.855633 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:28.855606    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:29.169781 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:29.169752    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"]
Oct 29 22:07:29.178857 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:07:29.178834    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b3220a5_a632_40bd_b9ad_668bb2b5b237.slice/crio-ef7be4397f63562a3ca111d5d1ca59dcd527bbd7064b251d48f5aa754bd9884d WatchSource:0}: Error finding container ef7be4397f63562a3ca111d5d1ca59dcd527bbd7064b251d48f5aa754bd9884d: Status 404 returned error can't find the container with id ef7be4397f63562a3ca111d5d1ca59dcd527bbd7064b251d48f5aa754bd9884d
Oct 29 22:07:29.181511 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:29.181479    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:07:29.181600 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:29.181542    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:07:29.181600 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:29.181569    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:07:29.216577 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:29.216539    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"]
Oct 29 22:07:29.220070 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:07:29.220043    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1106f45f_cdca_428f_8a2e_dc402d199ae8.slice/crio-91e07c2abdc914a1a77dfc96e2f9a6add574d2383b6b75455bbe51a3f9e16f3b WatchSource:0}: Error finding container 91e07c2abdc914a1a77dfc96e2f9a6add574d2383b6b75455bbe51a3f9e16f3b: Status 404 returned error can't find the container with id 91e07c2abdc914a1a77dfc96e2f9a6add574d2383b6b75455bbe51a3f9e16f3b
Oct 29 22:07:29.223523 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:29.223489    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:07:29.223612 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:29.223552    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:07:29.223612 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:29.223582    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:07:29.968082 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:29.968047    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg" event={"ID":"9b3220a5-a632-40bd-b9ad-668bb2b5b237","Type":"ContainerStarted","Data":"35a4dc9d31753702f3d8d75b201fa29ab3660c53a9bac13fbab6ee1c41d9a70e"}
Oct 29 22:07:29.968600 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:29.968126    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg" event={"ID":"9b3220a5-a632-40bd-b9ad-668bb2b5b237","Type":"ContainerStarted","Data":"ef7be4397f63562a3ca111d5d1ca59dcd527bbd7064b251d48f5aa754bd9884d"}
Oct 29 22:07:29.970006 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:29.969977    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk" event={"ID":"1106f45f-cdca-428f-8a2e-dc402d199ae8","Type":"ContainerStarted","Data":"2c1e96e174e66a2eab4f5e6502425730eae7015634f90fbf22e40dabb04b7ff0"}
Oct 29 22:07:29.970155 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:29.970020    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk" event={"ID":"1106f45f-cdca-428f-8a2e-dc402d199ae8","Type":"ContainerStarted","Data":"91e07c2abdc914a1a77dfc96e2f9a6add574d2383b6b75455bbe51a3f9e16f3b"}
Oct 29 22:07:30.026971 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:30.026920    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg" podStartSLOduration=2.026903296 podStartE2EDuration="2.026903296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:30.021455432 +0000 UTC m=+1271.094412408" watchObservedRunningTime="2026-10-29 22:07:30.026903296 +0000 UTC m=+1271.099860316"
Oct 29 22:07:30.071511 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:30.071462    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk" podStartSLOduration=2.071446583 podStartE2EDuration="2.071446583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:30.069435814 +0000 UTC m=+1271.142392777" watchObservedRunningTime="2026-10-29 22:07:30.071446583 +0000 UTC m=+1271.144403552"
Oct 29 22:07:30.832355 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:30.831834    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:30.833903 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:30.833872    4962 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 29 22:07:30.834133 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:30.834106    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg" podUID="9b3220a5-a632-40bd-b9ad-668bb2b5b237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 29 22:07:30.856247 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:30.856138    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:30.859071 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:30.859033    4962 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 29 22:07:30.859212 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:30.859100    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk" podUID="1106f45f-cdca-428f-8a2e-dc402d199ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 29 22:07:31.824514 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:31.823397    4962 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 29 22:07:31.824514 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:31.823443    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg" podUID="9b3220a5-a632-40bd-b9ad-668bb2b5b237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 29 22:07:31.855892 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:31.855861    4962 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 29 22:07:31.856037 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:31.855910    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk" podUID="1106f45f-cdca-428f-8a2e-dc402d199ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 29 22:07:33.154663 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:33.134266    4962 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 29 22:07:33.154663 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:33.134381    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg" podUID="9b3220a5-a632-40bd-b9ad-668bb2b5b237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 29 22:07:33.154663 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:33.136324    4962 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 29 22:07:33.154663 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:33.136372    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk" podUID="1106f45f-cdca-428f-8a2e-dc402d199ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 29 22:07:33.809329 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:33.801654    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:07:33.937365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:33.810000    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:34.021272 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:34.021224    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:07:34.031311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:34.030408    4962 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 29 22:07:34.031311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:34.030472    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg" podUID="9b3220a5-a632-40bd-b9ad-668bb2b5b237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 29 22:07:34.031311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:34.030923    4962 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 29 22:07:34.031311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:34.030968    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk" podUID="1106f45f-cdca-428f-8a2e-dc402d199ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 29 22:07:34.818582 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:34.817007    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9"]
Oct 29 22:07:34.825369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:34.824658    4962 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 29 22:07:34.825369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:34.824712    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg" podUID="9b3220a5-a632-40bd-b9ad-668bb2b5b237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 29 22:07:34.856862 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:34.856816    4962 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 29 22:07:34.856990 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:34.856879    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk" podUID="1106f45f-cdca-428f-8a2e-dc402d199ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 29 22:07:34.937024 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:34.936985    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22"]
Oct 29 22:07:35.824310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:35.823707    4962 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 29 22:07:35.824310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:35.823772    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg" podUID="9b3220a5-a632-40bd-b9ad-668bb2b5b237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 29 22:07:35.864678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:35.862691    4962 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 29 22:07:35.864678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:35.862753    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk" podUID="1106f45f-cdca-428f-8a2e-dc402d199ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 29 22:07:36.833315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:36.827828    4962 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 29 22:07:36.833315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:36.827888    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg" podUID="9b3220a5-a632-40bd-b9ad-668bb2b5b237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 29 22:07:36.859344 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:36.856015    4962 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 29 22:07:36.859344 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:36.856075    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk" podUID="1106f45f-cdca-428f-8a2e-dc402d199ae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 29 22:07:37.823467 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:37.823426    4962 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 29 22:07:37.823681 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:37.823482    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg" podUID="9b3220a5-a632-40bd-b9ad-668bb2b5b237" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 29 22:07:38.027574 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:38.027538    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:38.030580 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:38.028620    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:38.030935 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:38.030916    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:07:38.831361 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:38.831330    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:38.833353 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:38.833277    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:38.834959 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:38.834939    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:07:39.723422 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:07:39.723375    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:07:39.757164 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.757138    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"]
Oct 29 22:07:39.761901 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.761876    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.790647 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.790624    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"]
Oct 29 22:07:39.809387 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.809358    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ab68a7-4c08-494f-95c7-006502508963-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.809533 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.809390    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch7zs\" (UniqueName: \"kubernetes.io/projected/a8ab68a7-4c08-494f-95c7-006502508963-kube-api-access-ch7zs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.809533 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.809431    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.809644 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.809611    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ab68a7-4c08-494f-95c7-006502508963-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.809754 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.809735    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.809811 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.809799    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.809861 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.809825    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.809939 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.809923    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ab68a7-4c08-494f-95c7-006502508963-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.810007 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.809990    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.817369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.817340    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9" podUID="5ed4e4a6-8b44-4368-9194-47aa1bd10f0f" containerName="payload-processing" containerID="cri-o://4b1239a6dabc339cdef95c47598a1a68520e824235fdb8f89c4c2133849bcf53" gracePeriod=25
Oct 29 22:07:39.857940 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.857920    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-dd38c2-openshift-default-dockercfg-bqcds\""
Oct 29 22:07:39.911985 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.911953    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.912093 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.912053    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ab68a7-4c08-494f-95c7-006502508963-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.912157 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.912144    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.912218 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.912191    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.912218 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.912208    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.912342 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.912261    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ab68a7-4c08-494f-95c7-006502508963-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.912406 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.912391    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.912490 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.912469    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.912554 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.912514    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ab68a7-4c08-494f-95c7-006502508963-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.912554 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.912543    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch7zs\" (UniqueName: \"kubernetes.io/projected/a8ab68a7-4c08-494f-95c7-006502508963-kube-api-access-ch7zs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.912864 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.912824    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.912864 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.912854    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.913038 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.912891    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.913038 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.913020    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ab68a7-4c08-494f-95c7-006502508963-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.914796 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.914769    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.915237 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.915212    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ab68a7-4c08-494f-95c7-006502508963-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.937946 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.937921    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22" podUID="0ae786ab-5df2-457c-bbd9-93d6733cd7c4" containerName="payload-pre-processing" containerID="cri-o://ce340fc13f2c9634d1e6aa1cc5ad1defa0facaa94ee03760120409d3e6df913f" gracePeriod=25
Oct 29 22:07:39.941404 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.941378    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ab68a7-4c08-494f-95c7-006502508963-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:39.941850 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:39.941829    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch7zs\" (UniqueName: \"kubernetes.io/projected/a8ab68a7-4c08-494f-95c7-006502508963-kube-api-access-ch7zs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:40.091070 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.086263    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:40.119421 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.119397    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9"
Oct 29 22:07:40.226357 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.219571    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ed4e4a6-8b44-4368-9194-47aa1bd10f0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed4e4a6-8b44-4368-9194-47aa1bd10f0f-plugins-config-volume\") pod \"5ed4e4a6-8b44-4368-9194-47aa1bd10f0f\" (UID: \"5ed4e4a6-8b44-4368-9194-47aa1bd10f0f\") "
Oct 29 22:07:40.226357 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.219879    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ed4e4a6-8b44-4368-9194-47aa1bd10f0f-kube-api-access-m6vqn\" (UniqueName: \"kubernetes.io/projected/5ed4e4a6-8b44-4368-9194-47aa1bd10f0f-kube-api-access-m6vqn\") pod \"5ed4e4a6-8b44-4368-9194-47aa1bd10f0f\" (UID: \"5ed4e4a6-8b44-4368-9194-47aa1bd10f0f\") "
Oct 29 22:07:40.226357 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.221069    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ed4e4a6-8b44-4368-9194-47aa1bd10f0f-plugins-config-volume" pod "5ed4e4a6-8b44-4368-9194-47aa1bd10f0f" (UID: "5ed4e4a6-8b44-4368-9194-47aa1bd10f0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:07:40.226357 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.221315    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed4e4a6-8b44-4368-9194-47aa1bd10f0f-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:07:40.226357 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.223869    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed4e4a6-8b44-4368-9194-47aa1bd10f0f-kube-api-access-m6vqn" pod "5ed4e4a6-8b44-4368-9194-47aa1bd10f0f" (UID: "5ed4e4a6-8b44-4368-9194-47aa1bd10f0f"). InnerVolumeSpecName "kube-api-access-m6vqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:07:40.273131 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.273102    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22"
Oct 29 22:07:40.322749 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.322719    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ae786ab-5df2-457c-bbd9-93d6733cd7c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae786ab-5df2-457c-bbd9-93d6733cd7c4-plugins-config-volume\") pod \"0ae786ab-5df2-457c-bbd9-93d6733cd7c4\" (UID: \"0ae786ab-5df2-457c-bbd9-93d6733cd7c4\") "
Oct 29 22:07:40.323068 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.323045    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ae786ab-5df2-457c-bbd9-93d6733cd7c4-plugins-config-volume" pod "0ae786ab-5df2-457c-bbd9-93d6733cd7c4" (UID: "0ae786ab-5df2-457c-bbd9-93d6733cd7c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:07:40.323142 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.323123    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ae786ab-5df2-457c-bbd9-93d6733cd7c4-kube-api-access-m47m4\" (UniqueName: \"kubernetes.io/projected/0ae786ab-5df2-457c-bbd9-93d6733cd7c4-kube-api-access-m47m4\") pod \"0ae786ab-5df2-457c-bbd9-93d6733cd7c4\" (UID: \"0ae786ab-5df2-457c-bbd9-93d6733cd7c4\") "
Oct 29 22:07:40.324031 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.324004    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae786ab-5df2-457c-bbd9-93d6733cd7c4-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:07:40.324162 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.324034    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6vqn\" (UniqueName: \"kubernetes.io/projected/5ed4e4a6-8b44-4368-9194-47aa1bd10f0f-kube-api-access-m6vqn\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:07:40.325302 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.325267    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae786ab-5df2-457c-bbd9-93d6733cd7c4-kube-api-access-m47m4" pod "0ae786ab-5df2-457c-bbd9-93d6733cd7c4" (UID: "0ae786ab-5df2-457c-bbd9-93d6733cd7c4"). InnerVolumeSpecName "kube-api-access-m47m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:07:40.426076 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.426044    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m47m4\" (UniqueName: \"kubernetes.io/projected/0ae786ab-5df2-457c-bbd9-93d6733cd7c4-kube-api-access-m47m4\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:07:40.492501 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.492471    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"]
Oct 29 22:07:40.492631 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:07:40.492609    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ab68a7_4c08_494f_95c7_006502508963.slice/crio-23ebaabbc582d63d4a71ac1fe040ded52a3aceb73e783fd6c3d73bbbd67d5d5a WatchSource:0}: Error finding container 23ebaabbc582d63d4a71ac1fe040ded52a3aceb73e783fd6c3d73bbbd67d5d5a: Status 404 returned error can't find the container with id 23ebaabbc582d63d4a71ac1fe040ded52a3aceb73e783fd6c3d73bbbd67d5d5a
Oct 29 22:07:40.495396 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.495364    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:07:40.495514 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.495437    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:07:40.495514 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.495482    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:07:40.687042 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.687004    4962 generic.go:356] "Generic (PLEG): container finished" podID="0ae786ab-5df2-457c-bbd9-93d6733cd7c4" containerID="ce340fc13f2c9634d1e6aa1cc5ad1defa0facaa94ee03760120409d3e6df913f" exitCode=0
Oct 29 22:07:40.687239 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.687073    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22"
Oct 29 22:07:40.687239 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.687091    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22" event={"ID":"0ae786ab-5df2-457c-bbd9-93d6733cd7c4","Type":"ContainerDied","Data":"ce340fc13f2c9634d1e6aa1cc5ad1defa0facaa94ee03760120409d3e6df913f"}
Oct 29 22:07:40.687239 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.687183    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22" event={"ID":"0ae786ab-5df2-457c-bbd9-93d6733cd7c4","Type":"ContainerDied","Data":"d4f7a1dd6172b3ad084efe57db4e0b76715c46a942b057f1c318e56e9293814f"}
Oct 29 22:07:40.687239 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.687215    4962 scope.go:122] "RemoveContainer" containerID="ce340fc13f2c9634d1e6aa1cc5ad1defa0facaa94ee03760120409d3e6df913f"
Oct 29 22:07:40.689613 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.689591    4962 generic.go:356] "Generic (PLEG): container finished" podID="5ed4e4a6-8b44-4368-9194-47aa1bd10f0f" containerID="4b1239a6dabc339cdef95c47598a1a68520e824235fdb8f89c4c2133849bcf53" exitCode=0
Oct 29 22:07:40.689752 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.689726    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9" event={"ID":"5ed4e4a6-8b44-4368-9194-47aa1bd10f0f","Type":"ContainerDied","Data":"4b1239a6dabc339cdef95c47598a1a68520e824235fdb8f89c4c2133849bcf53"}
Oct 29 22:07:40.689829 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.689773    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9" event={"ID":"5ed4e4a6-8b44-4368-9194-47aa1bd10f0f","Type":"ContainerDied","Data":"d3a55dd5c2f8dab836a77197f03325bb118efa2430e9e20a0b5cd0cc0b531836"}
Oct 29 22:07:40.689894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.689829    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9"
Oct 29 22:07:40.693835 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.693747    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf" event={"ID":"a8ab68a7-4c08-494f-95c7-006502508963","Type":"ContainerStarted","Data":"146a99d6af0771756872912bf2cb97a8bcf63c3025bec77e33ac492b522fd026"}
Oct 29 22:07:40.693835 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.693790    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf" event={"ID":"a8ab68a7-4c08-494f-95c7-006502508963","Type":"ContainerStarted","Data":"23ebaabbc582d63d4a71ac1fe040ded52a3aceb73e783fd6c3d73bbbd67d5d5a"}
Oct 29 22:07:40.701129 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.701111    4962 scope.go:122] "RemoveContainer" containerID="ce340fc13f2c9634d1e6aa1cc5ad1defa0facaa94ee03760120409d3e6df913f"
Oct 29 22:07:40.701508 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:07:40.701483    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce340fc13f2c9634d1e6aa1cc5ad1defa0facaa94ee03760120409d3e6df913f\": container with ID starting with ce340fc13f2c9634d1e6aa1cc5ad1defa0facaa94ee03760120409d3e6df913f not found: ID does not exist" containerID="ce340fc13f2c9634d1e6aa1cc5ad1defa0facaa94ee03760120409d3e6df913f"
Oct 29 22:07:40.701596 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.701516    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce340fc13f2c9634d1e6aa1cc5ad1defa0facaa94ee03760120409d3e6df913f"} err="failed to get container status \"ce340fc13f2c9634d1e6aa1cc5ad1defa0facaa94ee03760120409d3e6df913f\": rpc error: code = NotFound desc = could not find container \"ce340fc13f2c9634d1e6aa1cc5ad1defa0facaa94ee03760120409d3e6df913f\": container with ID starting with ce340fc13f2c9634d1e6aa1cc5ad1defa0facaa94ee03760120409d3e6df913f not found: ID does not exist"
Oct 29 22:07:40.701596 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.701532    4962 scope.go:122] "RemoveContainer" containerID="4b1239a6dabc339cdef95c47598a1a68520e824235fdb8f89c4c2133849bcf53"
Oct 29 22:07:40.712516 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.712481    4962 scope.go:122] "RemoveContainer" containerID="4b1239a6dabc339cdef95c47598a1a68520e824235fdb8f89c4c2133849bcf53"
Oct 29 22:07:40.712818 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:07:40.712796    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b1239a6dabc339cdef95c47598a1a68520e824235fdb8f89c4c2133849bcf53\": container with ID starting with 4b1239a6dabc339cdef95c47598a1a68520e824235fdb8f89c4c2133849bcf53 not found: ID does not exist" containerID="4b1239a6dabc339cdef95c47598a1a68520e824235fdb8f89c4c2133849bcf53"
Oct 29 22:07:40.712932 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.712825    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b1239a6dabc339cdef95c47598a1a68520e824235fdb8f89c4c2133849bcf53"} err="failed to get container status \"4b1239a6dabc339cdef95c47598a1a68520e824235fdb8f89c4c2133849bcf53\": rpc error: code = NotFound desc = could not find container \"4b1239a6dabc339cdef95c47598a1a68520e824235fdb8f89c4c2133849bcf53\": container with ID starting with 4b1239a6dabc339cdef95c47598a1a68520e824235fdb8f89c4c2133849bcf53 not found: ID does not exist"
Oct 29 22:07:40.888842 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.888811    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9"]
Oct 29 22:07:40.908243 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.908208    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f363a3-766799c548-bxdc9"]
Oct 29 22:07:40.989186 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:40.989148    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22"]
Oct 29 22:07:41.002864 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:41.002830    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f363a3-66bd7467f9-mxf22"]
Oct 29 22:07:41.060699 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:41.060670    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ae786ab-5df2-457c-bbd9-93d6733cd7c4" path="/var/lib/kubelet/pods/0ae786ab-5df2-457c-bbd9-93d6733cd7c4/volumes"
Oct 29 22:07:41.061156 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:41.061140    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ed4e4a6-8b44-4368-9194-47aa1bd10f0f" path="/var/lib/kubelet/pods/5ed4e4a6-8b44-4368-9194-47aa1bd10f0f/volumes"
Oct 29 22:07:41.068517 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:41.068473    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf" podStartSLOduration=2.068459369 podStartE2EDuration="2.068459369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:41.067203751 +0000 UTC m=+1282.140160736" watchObservedRunningTime="2026-10-29 22:07:41.068459369 +0000 UTC m=+1282.141416332"
Oct 29 22:07:41.097365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:41.096573    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:41.101310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:41.100110    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 29 22:07:41.101310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:41.100195    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf" podUID="a8ab68a7-4c08-494f-95c7-006502508963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 29 22:07:42.096706 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:42.096633    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 29 22:07:42.097214 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:42.096774    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf" podUID="a8ab68a7-4c08-494f-95c7-006502508963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 29 22:07:43.095772 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.095737    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 29 22:07:43.096196 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.096165    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf" podUID="a8ab68a7-4c08-494f-95c7-006502508963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 29 22:07:43.609234 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.609202    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"]
Oct 29 22:07:43.610910 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.610887    4962 cpu_manager.go:414] "containerMap: removing container" podUID="0ae786ab-5df2-457c-bbd9-93d6733cd7c4" containerName="payload-pre-processing"
Oct 29 22:07:43.611056 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.611045    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ae786ab-5df2-457c-bbd9-93d6733cd7c4" containerName="payload-pre-processing"
Oct 29 22:07:43.611123 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.611116    4962 cpu_manager.go:414] "containerMap: removing container" podUID="5ed4e4a6-8b44-4368-9194-47aa1bd10f0f" containerName="payload-processing"
Oct 29 22:07:43.611177 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.611169    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ed4e4a6-8b44-4368-9194-47aa1bd10f0f" containerName="payload-processing"
Oct 29 22:07:43.611443 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.611425    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ae786ab-5df2-457c-bbd9-93d6733cd7c4" containerName="payload-pre-processing"
Oct 29 22:07:43.611558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.611545    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ed4e4a6-8b44-4368-9194-47aa1bd10f0f" containerName="payload-processing"
Oct 29 22:07:43.629454 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.627498    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.650745 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.643156    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"]
Oct 29 22:07:43.761605 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.761574    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a93430b5-4141-4183-a6f4-11297e0384c0-istio-token\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.761761 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.761635    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-credential-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.761854 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.761833    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-istio-envoy\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.761893 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.761864    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a93430b5-4141-4183-a6f4-11297e0384c0-istio-podinfo\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.761933 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.761896    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a93430b5-4141-4183-a6f4-11297e0384c0-istiod-ca-cert\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.762091 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.762076    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-istio-data\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.762127 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.762106    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-workload-certs\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.762158 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.762126    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrp49\" (UniqueName: \"kubernetes.io/projected/a93430b5-4141-4183-a6f4-11297e0384c0-kube-api-access-hrp49\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.762221 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.762210    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-workload-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.864009 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.863936    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-istio-data\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.864009 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.863978    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-workload-certs\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.864192 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.864008    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrp49\" (UniqueName: \"kubernetes.io/projected/a93430b5-4141-4183-a6f4-11297e0384c0-kube-api-access-hrp49\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.864192 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.864069    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-workload-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.864192 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.864141    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a93430b5-4141-4183-a6f4-11297e0384c0-istio-token\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.864408 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.864202    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-credential-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.864465 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.864408    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-istio-envoy\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.864465 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.864437    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a93430b5-4141-4183-a6f4-11297e0384c0-istio-podinfo\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.864567 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.864407    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-istio-data\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.864567 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.864475    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-workload-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.864567 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.864467    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a93430b5-4141-4183-a6f4-11297e0384c0-istiod-ca-cert\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.864668 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.864606    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-credential-socket\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.864742 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.864722    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-workload-certs\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.865162 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.865136    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a93430b5-4141-4183-a6f4-11297e0384c0-istiod-ca-cert\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.866747 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.866722    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-istio-envoy\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.866845 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.866829    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a93430b5-4141-4183-a6f4-11297e0384c0-istio-podinfo\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.895310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.893797    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a93430b5-4141-4183-a6f4-11297e0384c0-istio-token\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.898955 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.898927    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrp49\" (UniqueName: \"kubernetes.io/projected/a93430b5-4141-4183-a6f4-11297e0384c0-kube-api-access-hrp49\") pod \"e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") " pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:43.994370 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:43.988553    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:44.119844 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:44.112016    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 29 22:07:44.119844 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:44.112084    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf" podUID="a8ab68a7-4c08-494f-95c7-006502508963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 29 22:07:44.572196 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:44.571991    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"]
Oct 29 22:07:44.635173 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:44.635125    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:07:44.635608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:44.635210    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:07:44.635608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:44.635247    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:07:44.820826 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:44.820787    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25" event={"ID":"a93430b5-4141-4183-a6f4-11297e0384c0","Type":"ContainerStarted","Data":"1782df9d6fc7b77793e388f65492408da83206592cdea0cc34ebfa08d465e256"}
Oct 29 22:07:44.821017 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:44.820851    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25" event={"ID":"a93430b5-4141-4183-a6f4-11297e0384c0","Type":"ContainerStarted","Data":"ee572aee965b14926b5b23bf24a852b573f5e3390442bdd0ff8460122bdd6d8b"}
Oct 29 22:07:44.869082 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:44.868991    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25" podStartSLOduration=1.868977648 podStartE2EDuration="1.868977648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:44.867384568 +0000 UTC m=+1285.940341523" watchObservedRunningTime="2026-10-29 22:07:44.868977648 +0000 UTC m=+1285.941934612"
Oct 29 22:07:44.993310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:44.992399    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:45.098399 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:45.097597    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 29 22:07:45.098399 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:45.097654    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf" podUID="a8ab68a7-4c08-494f-95c7-006502508963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 29 22:07:46.007311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:46.006373    4962 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 29 22:07:46.007311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:46.006435    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25" podUID="a93430b5-4141-4183-a6f4-11297e0384c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 29 22:07:46.071150 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:46.071116    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"]
Oct 29 22:07:46.103310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:46.102515    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 29 22:07:46.103310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:46.102570    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf" podUID="a8ab68a7-4c08-494f-95c7-006502508963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 29 22:07:46.144950 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:46.129757    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"]
Oct 29 22:07:46.144950 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:46.129878    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"
Oct 29 22:07:46.151249 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:46.150132    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-df42c7-serving-cert\""
Oct 29 22:07:46.248514 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:46.248138    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bd99\" (UniqueName: \"kubernetes.io/projected/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-kube-api-access-7bd99\") pod \"maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt\" (UID: \"4c60f25d-1a16-4cbb-bd55-42cf7e13946e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"
Oct 29 22:07:46.249204 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:46.249175    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt\" (UID: \"4c60f25d-1a16-4cbb-bd55-42cf7e13946e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"
Oct 29 22:07:46.249360 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:46.249344    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-metrics-certs\") pod \"maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt\" (UID: \"4c60f25d-1a16-4cbb-bd55-42cf7e13946e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"
Oct 29 22:07:46.249433 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:46.249422    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt\" (UID: \"4c60f25d-1a16-4cbb-bd55-42cf7e13946e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"
Oct 29 22:07:46.368359 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:46.361869    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt\" (UID: \"4c60f25d-1a16-4cbb-bd55-42cf7e13946e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"
Oct 29 22:07:46.368359 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:46.362080    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-metrics-certs\") pod \"maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt\" (UID: \"4c60f25d-1a16-4cbb-bd55-42cf7e13946e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"
Oct 29 22:07:46.368359 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:46.362216    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt\" (UID: \"4c60f25d-1a16-4cbb-bd55-42cf7e13946e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"
Oct 29 22:07:46.368359 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:46.362605    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bd99\" (UniqueName: \"kubernetes.io/projected/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-kube-api-access-7bd99\") pod \"maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt\" (UID: \"4c60f25d-1a16-4cbb-bd55-42cf7e13946e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"
Oct 29 22:07:46.414643 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:07:46.412479    4962 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-serving-cert: secret "maas-api-e2e-ipp-b-df42c7-serving-cert" not found
Oct 29 22:07:46.414643 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:07:46.412582    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-maas-api-tls podName:4c60f25d-1a16-4cbb-bd55-42cf7e13946e nodeName:}" failed. No retries permitted until 2026-10-29 22:07:46.912557009 +0000 UTC m=+1287.985513954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-maas-api-tls") pod "maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt" (UID: "4c60f25d-1a16-4cbb-bd55-42cf7e13946e") : secret "maas-api-e2e-ipp-b-df42c7-serving-cert" not found
Oct 29 22:07:46.451187 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:46.451153    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt\" (UID: \"4c60f25d-1a16-4cbb-bd55-42cf7e13946e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"
Oct 29 22:07:46.655824 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:46.654950    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bd99\" (UniqueName: \"kubernetes.io/projected/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-kube-api-access-7bd99\") pod \"maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt\" (UID: \"4c60f25d-1a16-4cbb-bd55-42cf7e13946e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"
Oct 29 22:07:46.657117 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:46.656661    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-metrics-certs\") pod \"maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt\" (UID: \"4c60f25d-1a16-4cbb-bd55-42cf7e13946e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"
Oct 29 22:07:46.937633 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:46.937532    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt\" (UID: \"4c60f25d-1a16-4cbb-bd55-42cf7e13946e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"
Oct 29 22:07:46.940275 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:46.940242    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt\" (UID: \"4c60f25d-1a16-4cbb-bd55-42cf7e13946e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"
Oct 29 22:07:46.997762 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:46.997387    4962 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 29 22:07:46.997762 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:46.997452    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25" podUID="a93430b5-4141-4183-a6f4-11297e0384c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 29 22:07:47.059623 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:47.059579    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"
Oct 29 22:07:47.324258 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:47.323328    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:47.328382 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:47.328335    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:47.333877 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:47.333857    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:07:47.927483 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:47.927450    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"]
Oct 29 22:07:48.002381 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:47.996627    4962 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 29 22:07:48.002381 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:47.996688    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25" podUID="a93430b5-4141-4183-a6f4-11297e0384c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 29 22:07:48.932709 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:48.929224    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt" event={"ID":"4c60f25d-1a16-4cbb-bd55-42cf7e13946e","Type":"ContainerStarted","Data":"bcff9f095a02bf50417f7d2383986de72af5ce2bee19d478fc5df3df8ef9a124"}
Oct 29 22:07:48.932709 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:48.929279    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt" event={"ID":"4c60f25d-1a16-4cbb-bd55-42cf7e13946e","Type":"ContainerStarted","Data":"26342ad0be649d59c764a31abbec5360a796c92acc8e43b4099fa5f0e0eb3772"}
Oct 29 22:07:48.932709 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:48.929372    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"
Oct 29 22:07:48.990312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:48.990170    4962 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 29 22:07:48.990312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:48.990226    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25" podUID="a93430b5-4141-4183-a6f4-11297e0384c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 29 22:07:49.005700 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:49.005651    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt" podStartSLOduration=2.627640197 podStartE2EDuration="3.005639519s" totalImagesPullingTime="377.999322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:48.984723874 +0000 UTC m=+1290.057680861" watchObservedRunningTime="2026-10-29 22:07:49.005639519 +0000 UTC m=+1290.078596483"
Oct 29 22:07:49.990692 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:49.990653    4962 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 29 22:07:49.991316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:49.991262    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25" podUID="a93430b5-4141-4183-a6f4-11297e0384c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 29 22:07:50.303635 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:50.303562    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c"]
Oct 29 22:07:50.303844 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:50.303805    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c" podUID="a9d95970-4217-4ac1-ac2e-e05d38b10ea7" containerName="maas-api" containerID="cri-o://f084309b26d4b3411744737e42b935d3609a28f9c097711a0ed6dfe89f651ad0" gracePeriod=30
Oct 29 22:07:50.878558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:50.868942    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c"
Oct 29 22:07:51.002358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:50.998782    4962 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 29 22:07:51.002358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:50.998862    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25" podUID="a93430b5-4141-4183-a6f4-11297e0384c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 29 22:07:51.019946 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:51.018100    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-metrics-certs\") pod \"a9d95970-4217-4ac1-ac2e-e05d38b10ea7\" (UID: \"a9d95970-4217-4ac1-ac2e-e05d38b10ea7\") "
Oct 29 22:07:51.019946 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:51.018444    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-kube-api-access-f6z6n\" (UniqueName: \"kubernetes.io/projected/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-kube-api-access-f6z6n\") pod \"a9d95970-4217-4ac1-ac2e-e05d38b10ea7\" (UID: \"a9d95970-4217-4ac1-ac2e-e05d38b10ea7\") "
Oct 29 22:07:51.019946 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:51.018553    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-openshift-service-ca\") pod \"a9d95970-4217-4ac1-ac2e-e05d38b10ea7\" (UID: \"a9d95970-4217-4ac1-ac2e-e05d38b10ea7\") "
Oct 29 22:07:51.019946 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:51.018585    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-maas-api-tls\") pod \"a9d95970-4217-4ac1-ac2e-e05d38b10ea7\" (UID: \"a9d95970-4217-4ac1-ac2e-e05d38b10ea7\") "
Oct 29 22:07:51.046347 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:51.045871    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-maas-api-tls" pod "a9d95970-4217-4ac1-ac2e-e05d38b10ea7" (UID: "a9d95970-4217-4ac1-ac2e-e05d38b10ea7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:07:51.085706 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:51.085633    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-openshift-service-ca" pod "a9d95970-4217-4ac1-ac2e-e05d38b10ea7" (UID: "a9d95970-4217-4ac1-ac2e-e05d38b10ea7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:07:51.123310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:51.109002    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-metrics-certs" pod "a9d95970-4217-4ac1-ac2e-e05d38b10ea7" (UID: "a9d95970-4217-4ac1-ac2e-e05d38b10ea7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:07:51.150318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:51.130911    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:07:51.150318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:51.130945    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:07:51.150318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:51.130960    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:07:51.529455 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:51.495987    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-kube-api-access-f6z6n" pod "a9d95970-4217-4ac1-ac2e-e05d38b10ea7" (UID: "a9d95970-4217-4ac1-ac2e-e05d38b10ea7"). InnerVolumeSpecName "kube-api-access-f6z6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:07:51.533615 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:51.533581    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6z6n\" (UniqueName: \"kubernetes.io/projected/a9d95970-4217-4ac1-ac2e-e05d38b10ea7-kube-api-access-f6z6n\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:07:51.538977 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:51.538944    4962 generic.go:356] "Generic (PLEG): container finished" podID="a9d95970-4217-4ac1-ac2e-e05d38b10ea7" containerID="f084309b26d4b3411744737e42b935d3609a28f9c097711a0ed6dfe89f651ad0" exitCode=0
Oct 29 22:07:51.539700 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:51.539592    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c"
Oct 29 22:07:51.546229 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:51.546198    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c" event={"ID":"a9d95970-4217-4ac1-ac2e-e05d38b10ea7","Type":"ContainerDied","Data":"f084309b26d4b3411744737e42b935d3609a28f9c097711a0ed6dfe89f651ad0"}
Oct 29 22:07:51.546568 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:51.546259    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c" event={"ID":"a9d95970-4217-4ac1-ac2e-e05d38b10ea7","Type":"ContainerDied","Data":"19d6df5dba058a3fdd70644baf2ebd5499310bae1cbfc32ca9a3771d0e919a7d"}
Oct 29 22:07:51.546568 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:51.546314    4962 scope.go:122] "RemoveContainer" containerID="f084309b26d4b3411744737e42b935d3609a28f9c097711a0ed6dfe89f651ad0"
Oct 29 22:07:51.567505 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:51.567488    4962 scope.go:122] "RemoveContainer" containerID="f084309b26d4b3411744737e42b935d3609a28f9c097711a0ed6dfe89f651ad0"
Oct 29 22:07:51.567827 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:07:51.567809    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f084309b26d4b3411744737e42b935d3609a28f9c097711a0ed6dfe89f651ad0\": container with ID starting with f084309b26d4b3411744737e42b935d3609a28f9c097711a0ed6dfe89f651ad0 not found: ID does not exist" containerID="f084309b26d4b3411744737e42b935d3609a28f9c097711a0ed6dfe89f651ad0"
Oct 29 22:07:51.567957 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:51.567939    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f084309b26d4b3411744737e42b935d3609a28f9c097711a0ed6dfe89f651ad0"} err="failed to get container status \"f084309b26d4b3411744737e42b935d3609a28f9c097711a0ed6dfe89f651ad0\": rpc error: code = NotFound desc = could not find container \"f084309b26d4b3411744737e42b935d3609a28f9c097711a0ed6dfe89f651ad0\": container with ID starting with f084309b26d4b3411744737e42b935d3609a28f9c097711a0ed6dfe89f651ad0 not found: ID does not exist"
Oct 29 22:07:51.743596 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:51.743559    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c"]
Oct 29 22:07:51.752042 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:51.751804    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f6171d-58478bf69b-k6p4c"]
Oct 29 22:07:51.989944 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:51.989908    4962 patch_prober.go:29] interesting pod/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 29 22:07:51.990119 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:51.989983    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25" podUID="a93430b5-4141-4183-a6f4-11297e0384c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 29 22:07:52.993926 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:52.993894    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:52.994404 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:52.994387    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:52.994985 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:52.994964    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:07:53.061451 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:53.061423    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9d95970-4217-4ac1-ac2e-e05d38b10ea7" path="/var/lib/kubelet/pods/a9d95970-4217-4ac1-ac2e-e05d38b10ea7/volumes"
Oct 29 22:07:54.856109 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.856074    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"]
Oct 29 22:07:54.857224 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.857197    4962 cpu_manager.go:414] "containerMap: removing container" podUID="a9d95970-4217-4ac1-ac2e-e05d38b10ea7" containerName="maas-api"
Oct 29 22:07:54.857224 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.857222    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9d95970-4217-4ac1-ac2e-e05d38b10ea7" containerName="maas-api"
Oct 29 22:07:54.857443 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.857428    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9d95970-4217-4ac1-ac2e-e05d38b10ea7" containerName="maas-api"
Oct 29 22:07:54.863116 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.863099    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:54.875927 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.875882    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"]
Oct 29 22:07:54.880579 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.880558    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:54.889921 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.889276    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"]
Oct 29 22:07:54.930615 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.930578    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"]
Oct 29 22:07:54.945115 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.945075    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:54.945267 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.945126    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:54.945267 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.945163    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:54.945862 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.945341    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:54.945862 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.945562    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8p78\" (UniqueName: \"kubernetes.io/projected/b0573e45-68c1-4509-abae-91401555180e-kube-api-access-m8p78\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:54.945862 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.945635    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:54.945862 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.945671    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:54.945862 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.945697    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:54.945862 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.945794    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:54.946183 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.945896    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:54.946183 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.945945    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0573e45-68c1-4509-abae-91401555180e-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:54.946183 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.946031    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:54.946183 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.946103    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:54.946428 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.946239    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:54.946428 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.946245    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"
Oct 29 22:07:54.946428 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.946318    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0573e45-68c1-4509-abae-91401555180e-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:54.946428 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.946348    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:54.946428 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.946370    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wv4v\" (UniqueName: \"kubernetes.io/projected/977f25cb-ac5d-4290-a9a1-59322cc86f3d-kube-api-access-2wv4v\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:54.946428 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:54.946387    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0573e45-68c1-4509-abae-91401555180e-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:55.062310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.057715    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:55.064098 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.062582    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0573e45-68c1-4509-abae-91401555180e-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:55.064367 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.064342    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:55.081510 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.081463    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:55.081938 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.081902    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:55.082161 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.082142    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0573e45-68c1-4509-abae-91401555180e-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:55.082340 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.082321    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:55.082527 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.082498    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wv4v\" (UniqueName: \"kubernetes.io/projected/977f25cb-ac5d-4290-a9a1-59322cc86f3d-kube-api-access-2wv4v\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:55.082672 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.082653    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0573e45-68c1-4509-abae-91401555180e-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:55.098311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.082944    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:55.103310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.083038    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:55.103310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.084199    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:55.103310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.084562    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:55.103310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.085365    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0573e45-68c1-4509-abae-91401555180e-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:55.103310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.086108    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:55.103310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.098896    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:55.103310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.098941    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:55.103310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.098994    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:55.103310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.099185    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8p78\" (UniqueName: \"kubernetes.io/projected/b0573e45-68c1-4509-abae-91401555180e-kube-api-access-m8p78\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:55.103310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.100304    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:55.103310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.100364    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:55.103310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.100396    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:55.103310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.100442    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:55.103310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.101859    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:55.103310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.102471    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:55.103310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.102776    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:55.110432 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.110363    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:55.157319 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.156831    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:55.159508 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.159481    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:55.247403 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.223602    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0573e45-68c1-4509-abae-91401555180e-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:55.247403 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.247183    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8p78\" (UniqueName: \"kubernetes.io/projected/b0573e45-68c1-4509-abae-91401555180e-kube-api-access-m8p78\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:55.249904 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.249853    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:55.255748 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.250852    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0573e45-68c1-4509-abae-91401555180e-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:55.263514 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.263057    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:55.263514 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.263451    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:55.267078 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.267028    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wv4v\" (UniqueName: \"kubernetes.io/projected/977f25cb-ac5d-4290-a9a1-59322cc86f3d-kube-api-access-2wv4v\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:55.479621 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.479589    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:55.498374 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.498348    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:55.931465 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:07:55.931434    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977f25cb_ac5d_4290_a9a1_59322cc86f3d.slice/crio-4bf81f5f71b9bba76c70d6faeeeb98fd53d0438c0f2931811da4cc4f919b34ca WatchSource:0}: Error finding container 4bf81f5f71b9bba76c70d6faeeeb98fd53d0438c0f2931811da4cc4f919b34ca: Status 404 returned error can't find the container with id 4bf81f5f71b9bba76c70d6faeeeb98fd53d0438c0f2931811da4cc4f919b34ca
Oct 29 22:07:55.932666 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.932634    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"]
Oct 29 22:07:55.934158 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:07:55.934127    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0573e45_68c1_4509_abae_91401555180e.slice/crio-f247d92bdde14569d143219838c8b079336a5268ce003e845b4caf476f7823c3 WatchSource:0}: Error finding container f247d92bdde14569d143219838c8b079336a5268ce003e845b4caf476f7823c3: Status 404 returned error can't find the container with id f247d92bdde14569d143219838c8b079336a5268ce003e845b4caf476f7823c3
Oct 29 22:07:55.934802 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.934775    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:07:55.934902 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.934841    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:07:55.934902 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.934876    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:07:55.937221 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.937191    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:07:55.937333 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.937264    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:07:55.937392 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.937374    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:07:55.947691 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:55.947663    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"]
Oct 29 22:07:56.591200 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:56.591164    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d" event={"ID":"b0573e45-68c1-4509-abae-91401555180e","Type":"ContainerStarted","Data":"48a68d1dee50f13e4f72a6d12b111253b589e6af1052ac6ace6e4e22d806a140"}
Oct 29 22:07:56.591452 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:56.591220    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d" event={"ID":"b0573e45-68c1-4509-abae-91401555180e","Type":"ContainerStarted","Data":"f247d92bdde14569d143219838c8b079336a5268ce003e845b4caf476f7823c3"}
Oct 29 22:07:56.593227 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:56.593193    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb" event={"ID":"977f25cb-ac5d-4290-a9a1-59322cc86f3d","Type":"ContainerStarted","Data":"1fa8cbaced73613c14dc2976e215755a76950eeb60e721b1279ce533213e469d"}
Oct 29 22:07:56.593371 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:56.593246    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb" event={"ID":"977f25cb-ac5d-4290-a9a1-59322cc86f3d","Type":"ContainerStarted","Data":"4bf81f5f71b9bba76c70d6faeeeb98fd53d0438c0f2931811da4cc4f919b34ca"}
Oct 29 22:07:56.641877 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:56.641834    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d" podStartSLOduration=2.641820354 podStartE2EDuration="2.641820354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:56.638052548 +0000 UTC m=+1297.711009545" watchObservedRunningTime="2026-10-29 22:07:56.641820354 +0000 UTC m=+1297.714777318"
Oct 29 22:07:56.689365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:56.689301    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb" podStartSLOduration=2.689272808 podStartE2EDuration="2.689272808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:07:56.683339978 +0000 UTC m=+1297.756296941" watchObservedRunningTime="2026-10-29 22:07:56.689272808 +0000 UTC m=+1297.762229772"
Oct 29 22:07:57.490311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:57.486642    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:07:57.497459 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:57.497419    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 29 22:07:57.497693 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:57.497665    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb" podUID="977f25cb-ac5d-4290-a9a1-59322cc86f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 29 22:07:57.500683 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:57.500655    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:07:57.500944 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:57.500922    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 29 22:07:57.501069 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:57.501048    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d" podUID="b0573e45-68c1-4509-abae-91401555180e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 29 22:07:58.501310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:58.496076    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 29 22:07:58.501310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:58.496172    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb" podUID="977f25cb-ac5d-4290-a9a1-59322cc86f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 29 22:07:58.510308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:58.504404    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 29 22:07:58.510308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:58.504454    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d" podUID="b0573e45-68c1-4509-abae-91401555180e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 29 22:07:59.495312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:59.485629    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 29 22:07:59.495312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:59.485689    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb" podUID="977f25cb-ac5d-4290-a9a1-59322cc86f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 29 22:07:59.518312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:59.499640    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 29 22:07:59.518312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:07:59.499691    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d" podUID="b0573e45-68c1-4509-abae-91401555180e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 29 22:08:00.490373 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:00.487566    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 29 22:08:00.490373 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:00.487626    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb" podUID="977f25cb-ac5d-4290-a9a1-59322cc86f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 29 22:08:00.599354 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:00.598055    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 29 22:08:00.675310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:00.613787    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d" podUID="b0573e45-68c1-4509-abae-91401555180e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 29 22:08:01.539418 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:01.539379    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 29 22:08:01.539663 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:01.539440    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d" podUID="b0573e45-68c1-4509-abae-91401555180e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 29 22:08:01.546234 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:01.546202    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 29 22:08:01.546759 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:01.546271    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb" podUID="977f25cb-ac5d-4290-a9a1-59322cc86f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 29 22:08:02.486365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:02.483662    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 29 22:08:02.486365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:02.483726    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb" podUID="977f25cb-ac5d-4290-a9a1-59322cc86f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 29 22:08:02.513309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:02.511394    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 29 22:08:02.513309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:02.511471    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d" podUID="b0573e45-68c1-4509-abae-91401555180e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 29 22:08:03.485310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:03.481613    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 29 22:08:03.485310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:03.481689    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb" podUID="977f25cb-ac5d-4290-a9a1-59322cc86f3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 29 22:08:03.508191 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:03.508033    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 29 22:08:03.508191 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:03.508093    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d" podUID="b0573e45-68c1-4509-abae-91401555180e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 29 22:08:04.485391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:04.485361    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:08:04.485843 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:04.485814    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:08:04.486451 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:04.486435    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:08:04.502804 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:04.502783    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:08:04.503102 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:04.503081    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:08:04.503856 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:04.503830    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:08:06.212074 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:06.212039    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"]
Oct 29 22:08:06.212619 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:06.212270    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf" podUID="159da2ec-dc40-466b-ad09-0061dbd852a1" containerName="istio-proxy" containerID="cri-o://2fdaf48668c0a9160323b582044ad15e29c4079957e189b1c63115847eaefb49" gracePeriod=30
Oct 29 22:08:06.227635 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:06.227603    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"]
Oct 29 22:08:06.227845 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:06.227820    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw" podUID="728837f8-b92c-4025-b332-663f29e1570b" containerName="istio-proxy" containerID="cri-o://31e43a8fc4ee90eec49a8488fa44ccc527aefe226e7be0dde1a73caa54ba7745" gracePeriod=30
Oct 29 22:08:06.233187 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:06.233163    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"]
Oct 29 22:08:06.233382 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:06.233349    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4" podUID="8f0235b0-8935-415c-a47e-dd3ac5df7a84" containerName="istio-proxy" containerID="cri-o://43fe6a5d691049d163e10e716ebacc44ffe4835dcee189078bf88610948b9dce" gracePeriod=30
Oct 29 22:08:09.721980 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:08:09.721940    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:08:11.641388 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.641349    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:08:11.644108 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.644085    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:08:11.692555 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.692515    4962 generic.go:356] "Generic (PLEG): container finished" podID="728837f8-b92c-4025-b332-663f29e1570b" containerID="31e43a8fc4ee90eec49a8488fa44ccc527aefe226e7be0dde1a73caa54ba7745" exitCode=0
Oct 29 22:08:11.693139 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.692998    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"
Oct 29 22:08:11.693510 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.693018    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw" event={"ID":"728837f8-b92c-4025-b332-663f29e1570b","Type":"ContainerDied","Data":"31e43a8fc4ee90eec49a8488fa44ccc527aefe226e7be0dde1a73caa54ba7745"}
Oct 29 22:08:11.693696 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.693680    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw" event={"ID":"728837f8-b92c-4025-b332-663f29e1570b","Type":"ContainerDied","Data":"2480f00b1eaa05d888287efc332939fa50f1a71439b07809eef0ddb76e1e5499"}
Oct 29 22:08:11.693801 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.693791    4962 scope.go:122] "RemoveContainer" containerID="31e43a8fc4ee90eec49a8488fa44ccc527aefe226e7be0dde1a73caa54ba7745"
Oct 29 22:08:11.696888 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.696843    4962 generic.go:356] "Generic (PLEG): container finished" podID="159da2ec-dc40-466b-ad09-0061dbd852a1" containerID="2fdaf48668c0a9160323b582044ad15e29c4079957e189b1c63115847eaefb49" exitCode=0
Oct 29 22:08:11.697030 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.697008    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"
Oct 29 22:08:11.697326 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.697100    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf" event={"ID":"159da2ec-dc40-466b-ad09-0061dbd852a1","Type":"ContainerDied","Data":"2fdaf48668c0a9160323b582044ad15e29c4079957e189b1c63115847eaefb49"}
Oct 29 22:08:11.697478 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.697460    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf" event={"ID":"159da2ec-dc40-466b-ad09-0061dbd852a1","Type":"ContainerDied","Data":"30e143c9a253848df4a6749d118acb689ac04adac496633d61387c561ef4d911"}
Oct 29 22:08:11.699015 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.698990    4962 generic.go:356] "Generic (PLEG): container finished" podID="8f0235b0-8935-415c-a47e-dd3ac5df7a84" containerID="43fe6a5d691049d163e10e716ebacc44ffe4835dcee189078bf88610948b9dce" exitCode=0
Oct 29 22:08:11.699157 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.699141    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4" event={"ID":"8f0235b0-8935-415c-a47e-dd3ac5df7a84","Type":"ContainerDied","Data":"43fe6a5d691049d163e10e716ebacc44ffe4835dcee189078bf88610948b9dce"}
Oct 29 22:08:11.700668 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.700581    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:08:11.712204 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.711916    4962 scope.go:122] "RemoveContainer" containerID="31e43a8fc4ee90eec49a8488fa44ccc527aefe226e7be0dde1a73caa54ba7745"
Oct 29 22:08:11.712319 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:08:11.712218    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e43a8fc4ee90eec49a8488fa44ccc527aefe226e7be0dde1a73caa54ba7745\": container with ID starting with 31e43a8fc4ee90eec49a8488fa44ccc527aefe226e7be0dde1a73caa54ba7745 not found: ID does not exist" containerID="31e43a8fc4ee90eec49a8488fa44ccc527aefe226e7be0dde1a73caa54ba7745"
Oct 29 22:08:11.712319 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.712254    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e43a8fc4ee90eec49a8488fa44ccc527aefe226e7be0dde1a73caa54ba7745"} err="failed to get container status \"31e43a8fc4ee90eec49a8488fa44ccc527aefe226e7be0dde1a73caa54ba7745\": rpc error: code = NotFound desc = could not find container \"31e43a8fc4ee90eec49a8488fa44ccc527aefe226e7be0dde1a73caa54ba7745\": container with ID starting with 31e43a8fc4ee90eec49a8488fa44ccc527aefe226e7be0dde1a73caa54ba7745 not found: ID does not exist"
Oct 29 22:08:11.712319 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.712272    4962 scope.go:122] "RemoveContainer" containerID="2fdaf48668c0a9160323b582044ad15e29c4079957e189b1c63115847eaefb49"
Oct 29 22:08:11.726536 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.726510    4962 scope.go:122] "RemoveContainer" containerID="2fdaf48668c0a9160323b582044ad15e29c4079957e189b1c63115847eaefb49"
Oct 29 22:08:11.726821 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:08:11.726796    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fdaf48668c0a9160323b582044ad15e29c4079957e189b1c63115847eaefb49\": container with ID starting with 2fdaf48668c0a9160323b582044ad15e29c4079957e189b1c63115847eaefb49 not found: ID does not exist" containerID="2fdaf48668c0a9160323b582044ad15e29c4079957e189b1c63115847eaefb49"
Oct 29 22:08:11.726896 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.726833    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fdaf48668c0a9160323b582044ad15e29c4079957e189b1c63115847eaefb49"} err="failed to get container status \"2fdaf48668c0a9160323b582044ad15e29c4079957e189b1c63115847eaefb49\": rpc error: code = NotFound desc = could not find container \"2fdaf48668c0a9160323b582044ad15e29c4079957e189b1c63115847eaefb49\": container with ID starting with 2fdaf48668c0a9160323b582044ad15e29c4079957e189b1c63115847eaefb49 not found: ID does not exist"
Oct 29 22:08:11.728622 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.728599    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-podinfo\") pod \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") "
Oct 29 22:08:11.728718 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.728656    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-podinfo\") pod \"159da2ec-dc40-466b-ad09-0061dbd852a1\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") "
Oct 29 22:08:11.728761 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.728744    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-credential-socket\") pod \"728837f8-b92c-4025-b332-663f29e1570b\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") "
Oct 29 22:08:11.728814 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.728796    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/728837f8-b92c-4025-b332-663f29e1570b-istio-token\" (UniqueName: \"kubernetes.io/projected/728837f8-b92c-4025-b332-663f29e1570b-istio-token\") pod \"728837f8-b92c-4025-b332-663f29e1570b\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") "
Oct 29 22:08:11.728890 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.728855    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-envoy\") pod \"159da2ec-dc40-466b-ad09-0061dbd852a1\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") "
Oct 29 22:08:11.728964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.728926    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-workload-certs\") pod \"159da2ec-dc40-466b-ad09-0061dbd852a1\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") "
Oct 29 22:08:11.729021 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.728973    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-istio-data\") pod \"728837f8-b92c-4025-b332-663f29e1570b\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") "
Oct 29 22:08:11.729021 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.729002    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-credential-socket\") pod \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") "
Oct 29 22:08:11.729126 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.729058    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-workload-certs\") pod \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") "
Oct 29 22:08:11.729126 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.729097    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-data\") pod \"159da2ec-dc40-466b-ad09-0061dbd852a1\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") "
Oct 29 22:08:11.729243 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.729162    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/728837f8-b92c-4025-b332-663f29e1570b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728837f8-b92c-4025-b332-663f29e1570b-istiod-ca-cert\") pod \"728837f8-b92c-4025-b332-663f29e1570b\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") "
Oct 29 22:08:11.729243 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.729233    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-token\" (UniqueName: \"kubernetes.io/projected/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-token\") pod \"159da2ec-dc40-466b-ad09-0061dbd852a1\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") "
Oct 29 22:08:11.729415 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.729262    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/728837f8-b92c-4025-b332-663f29e1570b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728837f8-b92c-4025-b332-663f29e1570b-istio-podinfo\") pod \"728837f8-b92c-4025-b332-663f29e1570b\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") "
Oct 29 22:08:11.729415 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.729319    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/159da2ec-dc40-466b-ad09-0061dbd852a1-kube-api-access-t2n82\" (UniqueName: \"kubernetes.io/projected/159da2ec-dc40-466b-ad09-0061dbd852a1-kube-api-access-t2n82\") pod \"159da2ec-dc40-466b-ad09-0061dbd852a1\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") "
Oct 29 22:08:11.729415 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.729349    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istiod-ca-cert\") pod \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") "
Oct 29 22:08:11.729415 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.729400    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f0235b0-8935-415c-a47e-dd3ac5df7a84-kube-api-access-t77nk\" (UniqueName: \"kubernetes.io/projected/8f0235b0-8935-415c-a47e-dd3ac5df7a84-kube-api-access-t77nk\") pod \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") "
Oct 29 22:08:11.729634 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.729443    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/728837f8-b92c-4025-b332-663f29e1570b-kube-api-access-xwnnc\" (UniqueName: \"kubernetes.io/projected/728837f8-b92c-4025-b332-663f29e1570b-kube-api-access-xwnnc\") pod \"728837f8-b92c-4025-b332-663f29e1570b\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") "
Oct 29 22:08:11.729634 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.729544    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-istio-envoy\") pod \"728837f8-b92c-4025-b332-663f29e1570b\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") "
Oct 29 22:08:11.729634 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.729585    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-token\" (UniqueName: \"kubernetes.io/projected/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-token\") pod \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") "
Oct 29 22:08:11.729793 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.729654    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-workload-certs" pod "8f0235b0-8935-415c-a47e-dd3ac5df7a84" (UID: "8f0235b0-8935-415c-a47e-dd3ac5df7a84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.729793 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.729667    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-credential-socket\") pod \"159da2ec-dc40-466b-ad09-0061dbd852a1\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") "
Oct 29 22:08:11.729793 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.729728    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-envoy\") pod \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") "
Oct 29 22:08:11.729793 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.729763    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-workload-socket\") pod \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") "
Oct 29 22:08:11.729995 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.729817    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-workload-certs\") pod \"728837f8-b92c-4025-b332-663f29e1570b\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") "
Oct 29 22:08:11.729995 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.729904    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-credential-socket" pod "159da2ec-dc40-466b-ad09-0061dbd852a1" (UID: "159da2ec-dc40-466b-ad09-0061dbd852a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.729995 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.729978    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-workload-socket\") pod \"728837f8-b92c-4025-b332-663f29e1570b\" (UID: \"728837f8-b92c-4025-b332-663f29e1570b\") "
Oct 29 22:08:11.730174 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.730108    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-workload-certs" pod "728837f8-b92c-4025-b332-663f29e1570b" (UID: "728837f8-b92c-4025-b332-663f29e1570b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.730174 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.730116    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/159da2ec-dc40-466b-ad09-0061dbd852a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159da2ec-dc40-466b-ad09-0061dbd852a1-istiod-ca-cert\") pod \"159da2ec-dc40-466b-ad09-0061dbd852a1\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") "
Oct 29 22:08:11.730174 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.730157    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-workload-socket\") pod \"159da2ec-dc40-466b-ad09-0061dbd852a1\" (UID: \"159da2ec-dc40-466b-ad09-0061dbd852a1\") "
Oct 29 22:08:11.730365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.730342    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-data\") pod \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\" (UID: \"8f0235b0-8935-415c-a47e-dd3ac5df7a84\") "
Oct 29 22:08:11.730429 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.730414    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/159da2ec-dc40-466b-ad09-0061dbd852a1-istiod-ca-cert" pod "159da2ec-dc40-466b-ad09-0061dbd852a1" (UID: "159da2ec-dc40-466b-ad09-0061dbd852a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:11.730670 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.730649    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-data" pod "159da2ec-dc40-466b-ad09-0061dbd852a1" (UID: "159da2ec-dc40-466b-ad09-0061dbd852a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.730846 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.730828    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-data" pod "8f0235b0-8935-415c-a47e-dd3ac5df7a84" (UID: "8f0235b0-8935-415c-a47e-dd3ac5df7a84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.731168 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.731145    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/728837f8-b92c-4025-b332-663f29e1570b-istiod-ca-cert" pod "728837f8-b92c-4025-b332-663f29e1570b" (UID: "728837f8-b92c-4025-b332-663f29e1570b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:11.731481 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.731449    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.731481 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.731474    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.731620 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.731487    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/159da2ec-dc40-466b-ad09-0061dbd852a1-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.731620 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.731500    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.731620 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.731512    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.731620 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.731525    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.731620 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.731536    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728837f8-b92c-4025-b332-663f29e1570b-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.731885 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.731673    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-credential-socket" pod "728837f8-b92c-4025-b332-663f29e1570b" (UID: "728837f8-b92c-4025-b332-663f29e1570b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.734194 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.734169    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/728837f8-b92c-4025-b332-663f29e1570b-kube-api-access-xwnnc" pod "728837f8-b92c-4025-b332-663f29e1570b" (UID: "728837f8-b92c-4025-b332-663f29e1570b"). InnerVolumeSpecName "kube-api-access-xwnnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:11.734332 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.734220    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp"]
Oct 29 22:08:11.734415 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.734394    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-workload-socket" pod "728837f8-b92c-4025-b332-663f29e1570b" (UID: "728837f8-b92c-4025-b332-663f29e1570b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.734493 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.734475    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/728837f8-b92c-4025-b332-663f29e1570b-istio-token" pod "728837f8-b92c-4025-b332-663f29e1570b" (UID: "728837f8-b92c-4025-b332-663f29e1570b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:11.734605 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.734587    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-credential-socket" pod "8f0235b0-8935-415c-a47e-dd3ac5df7a84" (UID: "8f0235b0-8935-415c-a47e-dd3ac5df7a84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.736279 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.736254    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-envoy" pod "8f0235b0-8935-415c-a47e-dd3ac5df7a84" (UID: "8f0235b0-8935-415c-a47e-dd3ac5df7a84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.736434 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.736359    4962 cpu_manager.go:414] "containerMap: removing container" podUID="159da2ec-dc40-466b-ad09-0061dbd852a1" containerName="istio-proxy"
Oct 29 22:08:11.736434 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.736376    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="159da2ec-dc40-466b-ad09-0061dbd852a1" containerName="istio-proxy"
Oct 29 22:08:11.736434 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.736429    4962 cpu_manager.go:414] "containerMap: removing container" podUID="8f0235b0-8935-415c-a47e-dd3ac5df7a84" containerName="istio-proxy"
Oct 29 22:08:11.736608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.736437    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f0235b0-8935-415c-a47e-dd3ac5df7a84" containerName="istio-proxy"
Oct 29 22:08:11.736608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.736496    4962 cpu_manager.go:414] "containerMap: removing container" podUID="728837f8-b92c-4025-b332-663f29e1570b" containerName="istio-proxy"
Oct 29 22:08:11.736608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.736504    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="728837f8-b92c-4025-b332-663f29e1570b" containerName="istio-proxy"
Oct 29 22:08:11.736608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.736569    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-workload-socket" pod "8f0235b0-8935-415c-a47e-dd3ac5df7a84" (UID: "8f0235b0-8935-415c-a47e-dd3ac5df7a84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.736828 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.736713    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f0235b0-8935-415c-a47e-dd3ac5df7a84" containerName="istio-proxy"
Oct 29 22:08:11.736828 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.736738    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="728837f8-b92c-4025-b332-663f29e1570b" containerName="istio-proxy"
Oct 29 22:08:11.736828 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.736753    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="159da2ec-dc40-466b-ad09-0061dbd852a1" containerName="istio-proxy"
Oct 29 22:08:11.736828 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.736755    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-workload-socket" pod "159da2ec-dc40-466b-ad09-0061dbd852a1" (UID: "159da2ec-dc40-466b-ad09-0061dbd852a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.737036 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.736870    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-istio-envoy" pod "728837f8-b92c-4025-b332-663f29e1570b" (UID: "728837f8-b92c-4025-b332-663f29e1570b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.737388 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.737350    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-envoy" pod "159da2ec-dc40-466b-ad09-0061dbd852a1" (UID: "159da2ec-dc40-466b-ad09-0061dbd852a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.737639 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.737614    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-workload-certs" pod "159da2ec-dc40-466b-ad09-0061dbd852a1" (UID: "159da2ec-dc40-466b-ad09-0061dbd852a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.737825 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.737803    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-istio-data" pod "728837f8-b92c-4025-b332-663f29e1570b" (UID: "728837f8-b92c-4025-b332-663f29e1570b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:11.739140 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.739088    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/728837f8-b92c-4025-b332-663f29e1570b-istio-podinfo" pod "728837f8-b92c-4025-b332-663f29e1570b" (UID: "728837f8-b92c-4025-b332-663f29e1570b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:08:11.739543 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.739512    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istiod-ca-cert" pod "8f0235b0-8935-415c-a47e-dd3ac5df7a84" (UID: "8f0235b0-8935-415c-a47e-dd3ac5df7a84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:11.740111 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.740086    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159da2ec-dc40-466b-ad09-0061dbd852a1-kube-api-access-t2n82" pod "159da2ec-dc40-466b-ad09-0061dbd852a1" (UID: "159da2ec-dc40-466b-ad09-0061dbd852a1"). InnerVolumeSpecName "kube-api-access-t2n82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:11.743473 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.743452    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp"
Oct 29 22:08:11.748915 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.748038    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp"]
Oct 29 22:08:11.748915 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.748560    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-token" pod "159da2ec-dc40-466b-ad09-0061dbd852a1" (UID: "159da2ec-dc40-466b-ad09-0061dbd852a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:11.748915 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.748677    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-podinfo" pod "8f0235b0-8935-415c-a47e-dd3ac5df7a84" (UID: "8f0235b0-8935-415c-a47e-dd3ac5df7a84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:08:11.748915 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.748692    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-podinfo" pod "159da2ec-dc40-466b-ad09-0061dbd852a1" (UID: "159da2ec-dc40-466b-ad09-0061dbd852a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:08:11.748915 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.748726    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0235b0-8935-415c-a47e-dd3ac5df7a84-kube-api-access-t77nk" pod "8f0235b0-8935-415c-a47e-dd3ac5df7a84" (UID: "8f0235b0-8935-415c-a47e-dd3ac5df7a84"). InnerVolumeSpecName "kube-api-access-t77nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:11.751111 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.751081    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-token" pod "8f0235b0-8935-415c-a47e-dd3ac5df7a84" (UID: "8f0235b0-8935-415c-a47e-dd3ac5df7a84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:11.765864 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.765762    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f363a3\""
Oct 29 22:08:11.765864 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.765834    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f363a3-dockercfg-m55vq\""
Oct 29 22:08:11.833411 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.833335    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvrlr\" (UniqueName: \"kubernetes.io/projected/e08900e4-1f62-471e-b4ca-c5ee2df27d1b-kube-api-access-pvrlr\") pod \"payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp\" (UID: \"e08900e4-1f62-471e-b4ca-c5ee2df27d1b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp"
Oct 29 22:08:11.833411 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.833387    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e08900e4-1f62-471e-b4ca-c5ee2df27d1b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp\" (UID: \"e08900e4-1f62-471e-b4ca-c5ee2df27d1b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp"
Oct 29 22:08:11.833890 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.833860    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.834003 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.833896    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.834003 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.833912    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.834003 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.833926    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.834003 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.833939    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.834003 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.833953    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728837f8-b92c-4025-b332-663f29e1570b-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.834003 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.833967    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.834003 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.833981    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/159da2ec-dc40-466b-ad09-0061dbd852a1-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.834003 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.833993    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.834460 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.834007    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.834460 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.834021    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/159da2ec-dc40-466b-ad09-0061dbd852a1-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.834460 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.834033    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728837f8-b92c-4025-b332-663f29e1570b-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.834460 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.834046    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2n82\" (UniqueName: \"kubernetes.io/projected/159da2ec-dc40-466b-ad09-0061dbd852a1-kube-api-access-t2n82\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.834460 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.834059    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.834460 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.834072    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t77nk\" (UniqueName: \"kubernetes.io/projected/8f0235b0-8935-415c-a47e-dd3ac5df7a84-kube-api-access-t77nk\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.834460 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.834085    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwnnc\" (UniqueName: \"kubernetes.io/projected/728837f8-b92c-4025-b332-663f29e1570b-kube-api-access-xwnnc\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.834460 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.834099    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728837f8-b92c-4025-b332-663f29e1570b-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.834460 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.834112    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.834460 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.834124    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.834460 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.834137    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f0235b0-8935-415c-a47e-dd3ac5df7a84-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:11.935329 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.935256    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvrlr\" (UniqueName: \"kubernetes.io/projected/e08900e4-1f62-471e-b4ca-c5ee2df27d1b-kube-api-access-pvrlr\") pod \"payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp\" (UID: \"e08900e4-1f62-471e-b4ca-c5ee2df27d1b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp"
Oct 29 22:08:11.935487 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.935362    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e08900e4-1f62-471e-b4ca-c5ee2df27d1b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp\" (UID: \"e08900e4-1f62-471e-b4ca-c5ee2df27d1b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp"
Oct 29 22:08:11.936036 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.936016    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e08900e4-1f62-471e-b4ca-c5ee2df27d1b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp\" (UID: \"e08900e4-1f62-471e-b4ca-c5ee2df27d1b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp"
Oct 29 22:08:11.958984 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:11.958957    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvrlr\" (UniqueName: \"kubernetes.io/projected/e08900e4-1f62-471e-b4ca-c5ee2df27d1b-kube-api-access-pvrlr\") pod \"payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp\" (UID: \"e08900e4-1f62-471e-b4ca-c5ee2df27d1b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp"
Oct 29 22:08:12.043364 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:12.043335    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"]
Oct 29 22:08:12.051419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:12.051353    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-dkstw"]
Oct 29 22:08:12.066855 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:12.066829    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp"
Oct 29 22:08:12.088529 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:12.088501    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"]
Oct 29 22:08:12.096672 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:12.096602    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-sqzlf"]
Oct 29 22:08:12.515485 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:12.515459    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp"]
Oct 29 22:08:12.516016 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:08:12.515987    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08900e4_1f62_471e_b4ca_c5ee2df27d1b.slice/crio-48f4952d5447434492fba7475d2fb8dcaa6e437df01ad612ad3b11f69b19cb89 WatchSource:0}: Error finding container 48f4952d5447434492fba7475d2fb8dcaa6e437df01ad612ad3b11f69b19cb89: Status 404 returned error can't find the container with id 48f4952d5447434492fba7475d2fb8dcaa6e437df01ad612ad3b11f69b19cb89
Oct 29 22:08:12.830518 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:12.817707    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp" event={"ID":"e08900e4-1f62-471e-b4ca-c5ee2df27d1b","Type":"ContainerStarted","Data":"48f4952d5447434492fba7475d2fb8dcaa6e437df01ad612ad3b11f69b19cb89"}
Oct 29 22:08:12.842437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:12.835575    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4" event={"ID":"8f0235b0-8935-415c-a47e-dd3ac5df7a84","Type":"ContainerDied","Data":"94e1afb28b82395174586d2bad5b08052562e5cf29ee8b20a6e2441382684598"}
Oct 29 22:08:12.842437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:12.835658    4962 scope.go:122] "RemoveContainer" containerID="43fe6a5d691049d163e10e716ebacc44ffe4835dcee189078bf88610948b9dce"
Oct 29 22:08:12.842437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:12.835818    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"
Oct 29 22:08:12.942016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:12.941982    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"]
Oct 29 22:08:12.952479 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:12.952450    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-751a7416-openshift-default-7cf959db5-xcpv4"]
Oct 29 22:08:13.064647 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:13.064611    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="159da2ec-dc40-466b-ad09-0061dbd852a1" path="/var/lib/kubelet/pods/159da2ec-dc40-466b-ad09-0061dbd852a1/volumes"
Oct 29 22:08:13.065443 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:13.065420    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="728837f8-b92c-4025-b332-663f29e1570b" path="/var/lib/kubelet/pods/728837f8-b92c-4025-b332-663f29e1570b/volumes"
Oct 29 22:08:13.066331 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:13.066138    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f0235b0-8935-415c-a47e-dd3ac5df7a84" path="/var/lib/kubelet/pods/8f0235b0-8935-415c-a47e-dd3ac5df7a84/volumes"
Oct 29 22:08:14.859466 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:14.859382    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp" event={"ID":"e08900e4-1f62-471e-b4ca-c5ee2df27d1b","Type":"ContainerStarted","Data":"58bb02b6dc61c4de0bbcd389431bfe229b2ba69d17be127bfdf731a3e4fc36a9"}
Oct 29 22:08:14.860118 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:14.859599    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp"
Oct 29 22:08:14.908178 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:14.908131    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp" podStartSLOduration=3.851067937 podStartE2EDuration="5.908117485s" totalImagesPullingTime="2.057049548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:14.902206666 +0000 UTC m=+1315.975163641" watchObservedRunningTime="2026-10-29 22:08:14.908117485 +0000 UTC m=+1315.981074449"
Oct 29 22:08:17.519217 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:17.519178    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf"]
Oct 29 22:08:17.525625 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:17.525603    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf"
Oct 29 22:08:17.540478 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:17.540451    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9ae129\""
Oct 29 22:08:17.541886 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:17.541861    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9ae129-dockercfg-gk5tc\""
Oct 29 22:08:17.547903 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:17.547879    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf"]
Oct 29 22:08:17.753306 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:17.752516    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50625ced-c86d-42a4-ab2d-7cff60f95ce6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf\" (UID: \"50625ced-c86d-42a4-ab2d-7cff60f95ce6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf"
Oct 29 22:08:17.753306 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:17.752784    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rflwb\" (UniqueName: \"kubernetes.io/projected/50625ced-c86d-42a4-ab2d-7cff60f95ce6-kube-api-access-rflwb\") pod \"payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf\" (UID: \"50625ced-c86d-42a4-ab2d-7cff60f95ce6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf"
Oct 29 22:08:17.836369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:17.831717    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4"]
Oct 29 22:08:17.858315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:17.855342    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50625ced-c86d-42a4-ab2d-7cff60f95ce6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf\" (UID: \"50625ced-c86d-42a4-ab2d-7cff60f95ce6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf"
Oct 29 22:08:17.858315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:17.855440    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rflwb\" (UniqueName: \"kubernetes.io/projected/50625ced-c86d-42a4-ab2d-7cff60f95ce6-kube-api-access-rflwb\") pod \"payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf\" (UID: \"50625ced-c86d-42a4-ab2d-7cff60f95ce6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf"
Oct 29 22:08:17.888311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:17.877973    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50625ced-c86d-42a4-ab2d-7cff60f95ce6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf\" (UID: \"50625ced-c86d-42a4-ab2d-7cff60f95ce6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf"
Oct 29 22:08:17.945909 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:17.940036    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rflwb\" (UniqueName: \"kubernetes.io/projected/50625ced-c86d-42a4-ab2d-7cff60f95ce6-kube-api-access-rflwb\") pod \"payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf\" (UID: \"50625ced-c86d-42a4-ab2d-7cff60f95ce6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf"
Oct 29 22:08:17.969310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:17.961467    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4"
Oct 29 22:08:17.969310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:17.965067    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4"]
Oct 29 22:08:18.060251 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:18.060219    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj564\" (UniqueName: \"kubernetes.io/projected/81379dd3-6066-452f-868b-8bde1172bd56-kube-api-access-tj564\") pod \"payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4\" (UID: \"81379dd3-6066-452f-868b-8bde1172bd56\") " pod="openshift-ingress/payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4"
Oct 29 22:08:18.060526 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:18.060512    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81379dd3-6066-452f-868b-8bde1172bd56-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4\" (UID: \"81379dd3-6066-452f-868b-8bde1172bd56\") " pod="openshift-ingress/payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4"
Oct 29 22:08:18.142908 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:18.142833    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf"
Oct 29 22:08:18.161705 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:18.161677    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81379dd3-6066-452f-868b-8bde1172bd56-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4\" (UID: \"81379dd3-6066-452f-868b-8bde1172bd56\") " pod="openshift-ingress/payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4"
Oct 29 22:08:18.161841 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:18.161828    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj564\" (UniqueName: \"kubernetes.io/projected/81379dd3-6066-452f-868b-8bde1172bd56-kube-api-access-tj564\") pod \"payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4\" (UID: \"81379dd3-6066-452f-868b-8bde1172bd56\") " pod="openshift-ingress/payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4"
Oct 29 22:08:18.162404 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:18.162374    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81379dd3-6066-452f-868b-8bde1172bd56-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4\" (UID: \"81379dd3-6066-452f-868b-8bde1172bd56\") " pod="openshift-ingress/payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4"
Oct 29 22:08:18.185500 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:18.185473    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj564\" (UniqueName: \"kubernetes.io/projected/81379dd3-6066-452f-868b-8bde1172bd56-kube-api-access-tj564\") pod \"payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4\" (UID: \"81379dd3-6066-452f-868b-8bde1172bd56\") " pod="openshift-ingress/payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4"
Oct 29 22:08:18.308263 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:18.308229    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4"
Oct 29 22:08:18.463458 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:18.462825    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf"]
Oct 29 22:08:18.656542 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:08:18.656510    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81379dd3_6066_452f_868b_8bde1172bd56.slice/crio-ee1f51baa788fb14a8c91453e1a8a07d5f10e10f249941965e57dfe793762124 WatchSource:0}: Error finding container ee1f51baa788fb14a8c91453e1a8a07d5f10e10f249941965e57dfe793762124: Status 404 returned error can't find the container with id ee1f51baa788fb14a8c91453e1a8a07d5f10e10f249941965e57dfe793762124
Oct 29 22:08:18.657332 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:18.657305    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4"]
Oct 29 22:08:18.948112 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:18.948074    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4" event={"ID":"81379dd3-6066-452f-868b-8bde1172bd56","Type":"ContainerStarted","Data":"c1d1488819b8ceccb0b0a3e43831aabc3b681bac78198ff880f12901365eb62f"}
Oct 29 22:08:18.948281 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:18.948138    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4" event={"ID":"81379dd3-6066-452f-868b-8bde1172bd56","Type":"ContainerStarted","Data":"ee1f51baa788fb14a8c91453e1a8a07d5f10e10f249941965e57dfe793762124"}
Oct 29 22:08:18.948281 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:18.948169    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4"
Oct 29 22:08:18.949993 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:18.949965    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf" event={"ID":"50625ced-c86d-42a4-ab2d-7cff60f95ce6","Type":"ContainerStarted","Data":"a75a1cfbf825b281245b6fce21c42f605fca5811d5a644ffd276681440cbfbf1"}
Oct 29 22:08:18.950109 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:18.950017    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf" event={"ID":"50625ced-c86d-42a4-ab2d-7cff60f95ce6","Type":"ContainerStarted","Data":"cf2cfef6d61fe0838449cbb3aae02622372a87d413b14d79a6f067d9a889da82"}
Oct 29 22:08:18.950109 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:18.950082    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf"
Oct 29 22:08:18.989081 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:18.989000    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4" podStartSLOduration=1.988988307 podStartE2EDuration="1.988988307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:18.987979117 +0000 UTC m=+1320.060936083" watchObservedRunningTime="2026-10-29 22:08:18.988988307 +0000 UTC m=+1320.061945272"
Oct 29 22:08:19.031870 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:19.031820    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf" podStartSLOduration=2.031807885 podStartE2EDuration="2.031807885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:19.028269952 +0000 UTC m=+1320.101226916" watchObservedRunningTime="2026-10-29 22:08:19.031807885 +0000 UTC m=+1320.104764848"
Oct 29 22:08:21.010821 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:21.010736    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847"]
Oct 29 22:08:21.043570 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:21.043537    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n"]
Oct 29 22:08:25.076429 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:25.076397    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn"]
Oct 29 22:08:25.109011 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:25.108985    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9"]
Oct 29 22:08:25.911398 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:25.908508    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp"
Oct 29 22:08:26.045271 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:26.040257    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847" podUID="e6fe16e8-3928-4766-9daf-7a449c61233d" containerName="payload-processing" containerID="cri-o://6ee7b4fb3db237dea450cb090ad407e84a85183c300362d3b8070fed57aac232" gracePeriod=25
Oct 29 22:08:26.045271 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:26.045155    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n" podUID="c2f96700-2d4a-4b30-ada0-04cf70b7f6d2" containerName="payload-pre-processing" containerID="cri-o://e8b9f21452e1283dc320b501e5470c07c9a9247e2d2b7535392d8912fd9a38e9" gracePeriod=25
Oct 29 22:08:26.224009 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:26.223974    4962 generic.go:356] "Generic (PLEG): container finished" podID="e6fe16e8-3928-4766-9daf-7a449c61233d" containerID="6ee7b4fb3db237dea450cb090ad407e84a85183c300362d3b8070fed57aac232" exitCode=0
Oct 29 22:08:26.224540 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:26.224210    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847" event={"ID":"e6fe16e8-3928-4766-9daf-7a449c61233d","Type":"ContainerDied","Data":"6ee7b4fb3db237dea450cb090ad407e84a85183c300362d3b8070fed57aac232"}
Oct 29 22:08:26.226531 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:26.226503    4962 generic.go:356] "Generic (PLEG): container finished" podID="c2f96700-2d4a-4b30-ada0-04cf70b7f6d2" containerID="e8b9f21452e1283dc320b501e5470c07c9a9247e2d2b7535392d8912fd9a38e9" exitCode=0
Oct 29 22:08:26.226667 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:26.226643    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n" event={"ID":"c2f96700-2d4a-4b30-ada0-04cf70b7f6d2","Type":"ContainerDied","Data":"e8b9f21452e1283dc320b501e5470c07c9a9247e2d2b7535392d8912fd9a38e9"}
Oct 29 22:08:26.410037 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:26.410006    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n"
Oct 29 22:08:26.532354 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:26.491512    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2f96700-2d4a-4b30-ada0-04cf70b7f6d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f96700-2d4a-4b30-ada0-04cf70b7f6d2-plugins-config-volume\") pod \"c2f96700-2d4a-4b30-ada0-04cf70b7f6d2\" (UID: \"c2f96700-2d4a-4b30-ada0-04cf70b7f6d2\") "
Oct 29 22:08:26.532354 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:26.491683    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2f96700-2d4a-4b30-ada0-04cf70b7f6d2-kube-api-access-x8lrn\" (UniqueName: \"kubernetes.io/projected/c2f96700-2d4a-4b30-ada0-04cf70b7f6d2-kube-api-access-x8lrn\") pod \"c2f96700-2d4a-4b30-ada0-04cf70b7f6d2\" (UID: \"c2f96700-2d4a-4b30-ada0-04cf70b7f6d2\") "
Oct 29 22:08:26.532354 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:26.493066    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2f96700-2d4a-4b30-ada0-04cf70b7f6d2-plugins-config-volume" pod "c2f96700-2d4a-4b30-ada0-04cf70b7f6d2" (UID: "c2f96700-2d4a-4b30-ada0-04cf70b7f6d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:26.532354 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:26.504396    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f96700-2d4a-4b30-ada0-04cf70b7f6d2-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:26.534047 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:26.534007    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2f96700-2d4a-4b30-ada0-04cf70b7f6d2-kube-api-access-x8lrn" pod "c2f96700-2d4a-4b30-ada0-04cf70b7f6d2" (UID: "c2f96700-2d4a-4b30-ada0-04cf70b7f6d2"). InnerVolumeSpecName "kube-api-access-x8lrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:26.538416 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:26.538017    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847"
Oct 29 22:08:26.609373 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:26.607718    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8lrn\" (UniqueName: \"kubernetes.io/projected/c2f96700-2d4a-4b30-ada0-04cf70b7f6d2-kube-api-access-x8lrn\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:26.709247 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:26.709218    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6fe16e8-3928-4766-9daf-7a449c61233d-kube-api-access-7x72r\" (UniqueName: \"kubernetes.io/projected/e6fe16e8-3928-4766-9daf-7a449c61233d-kube-api-access-7x72r\") pod \"e6fe16e8-3928-4766-9daf-7a449c61233d\" (UID: \"e6fe16e8-3928-4766-9daf-7a449c61233d\") "
Oct 29 22:08:26.709489 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:26.709446    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6fe16e8-3928-4766-9daf-7a449c61233d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6fe16e8-3928-4766-9daf-7a449c61233d-plugins-config-volume\") pod \"e6fe16e8-3928-4766-9daf-7a449c61233d\" (UID: \"e6fe16e8-3928-4766-9daf-7a449c61233d\") "
Oct 29 22:08:26.709866 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:26.709821    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6fe16e8-3928-4766-9daf-7a449c61233d-plugins-config-volume" pod "e6fe16e8-3928-4766-9daf-7a449c61233d" (UID: "e6fe16e8-3928-4766-9daf-7a449c61233d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:26.710213 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:26.710195    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6fe16e8-3928-4766-9daf-7a449c61233d-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:26.711503 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:26.711481    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6fe16e8-3928-4766-9daf-7a449c61233d-kube-api-access-7x72r" pod "e6fe16e8-3928-4766-9daf-7a449c61233d" (UID: "e6fe16e8-3928-4766-9daf-7a449c61233d"). InnerVolumeSpecName "kube-api-access-7x72r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:26.812181 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:26.812091    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x72r\" (UniqueName: \"kubernetes.io/projected/e6fe16e8-3928-4766-9daf-7a449c61233d-kube-api-access-7x72r\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:27.236106 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:27.236073    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847"
Oct 29 22:08:27.236552 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:27.236105    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847" event={"ID":"e6fe16e8-3928-4766-9daf-7a449c61233d","Type":"ContainerDied","Data":"739c5ac37be0f04cfd8ae45f69ccd695b7b57565844dbb01855090fcd69890c9"}
Oct 29 22:08:27.236552 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:27.236167    4962 scope.go:122] "RemoveContainer" containerID="6ee7b4fb3db237dea450cb090ad407e84a85183c300362d3b8070fed57aac232"
Oct 29 22:08:27.237694 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:27.237665    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n" event={"ID":"c2f96700-2d4a-4b30-ada0-04cf70b7f6d2","Type":"ContainerDied","Data":"e98c4fada6bebd9b4d42317d0d436b222371ff6d4cf11492f18e54825143b54b"}
Oct 29 22:08:27.237694 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:27.237691    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n"
Oct 29 22:08:27.247387 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:27.247366    4962 scope.go:122] "RemoveContainer" containerID="e8b9f21452e1283dc320b501e5470c07c9a9247e2d2b7535392d8912fd9a38e9"
Oct 29 22:08:27.295807 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:27.295779    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847"]
Oct 29 22:08:27.308376 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:27.308350    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-edc2d3-5cc66df658-x7847"]
Oct 29 22:08:27.333225 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:27.333158    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n"]
Oct 29 22:08:27.343363 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:27.343332    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-edc2d3-5fb66bfbcf-tgh5n"]
Oct 29 22:08:28.184337 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.182273    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"]
Oct 29 22:08:28.184337 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.182545    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x" podUID="fb5e9803-65bc-4b86-815b-3f1bc323b3bb" containerName="maas-api" containerID="cri-o://4b55130cebce2b54497dd37035b435da89cc38491b19adb1802ce1d15fbf644e" gracePeriod=30
Oct 29 22:08:28.277221 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.277187    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"]
Oct 29 22:08:28.278538 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.278514    4962 cpu_manager.go:414] "containerMap: removing container" podUID="c2f96700-2d4a-4b30-ada0-04cf70b7f6d2" containerName="payload-pre-processing"
Oct 29 22:08:28.278610 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.278541    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2f96700-2d4a-4b30-ada0-04cf70b7f6d2" containerName="payload-pre-processing"
Oct 29 22:08:28.278680 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.278637    4962 cpu_manager.go:414] "containerMap: removing container" podUID="e6fe16e8-3928-4766-9daf-7a449c61233d" containerName="payload-processing"
Oct 29 22:08:28.278680 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.278646    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6fe16e8-3928-4766-9daf-7a449c61233d" containerName="payload-processing"
Oct 29 22:08:28.278882 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.278862    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6fe16e8-3928-4766-9daf-7a449c61233d" containerName="payload-processing"
Oct 29 22:08:28.279055 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.279038    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2f96700-2d4a-4b30-ada0-04cf70b7f6d2" containerName="payload-pre-processing"
Oct 29 22:08:28.283586 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.283565    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.303490 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.303465    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"]
Oct 29 22:08:28.304544 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.304488    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a9e67f-openshift-default-dockercfg-kbmzv\""
Oct 29 22:08:28.340371 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.340334    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-workload-certs\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.340578 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.340387    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bced75a3-37ba-4595-9dc0-b354574eba07-istiod-ca-cert\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.340578 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.340432    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-credential-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.340578 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.340490    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbj75\" (UniqueName: \"kubernetes.io/projected/bced75a3-37ba-4595-9dc0-b354574eba07-kube-api-access-jbj75\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.342470 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.342440    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-istio-data\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.342607 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.342494    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-workload-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.342673 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.342623    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-istio-envoy\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.342894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.342727    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bced75a3-37ba-4595-9dc0-b354574eba07-istio-token\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.342894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.342785    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bced75a3-37ba-4595-9dc0-b354574eba07-istio-podinfo\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.444852 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.444774    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-istio-data\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.444852 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.444835    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-workload-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.445029 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.444970    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-istio-envoy\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.445116 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.445088    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bced75a3-37ba-4595-9dc0-b354574eba07-istio-token\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.445157 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.445144    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bced75a3-37ba-4595-9dc0-b354574eba07-istio-podinfo\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.445326 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.445301    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-istio-data\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.445326 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.445314    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-workload-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.445431 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.445391    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-workload-certs\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.445431 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.445416    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bced75a3-37ba-4595-9dc0-b354574eba07-istiod-ca-cert\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.445537 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.445448    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-credential-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.445537 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.445495    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbj75\" (UniqueName: \"kubernetes.io/projected/bced75a3-37ba-4595-9dc0-b354574eba07-kube-api-access-jbj75\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.446018 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.445993    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-credential-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.446320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.446255    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-workload-certs\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.446320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.446303    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bced75a3-37ba-4595-9dc0-b354574eba07-istiod-ca-cert\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.447840 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.447817    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-istio-envoy\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.448208 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.448170    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bced75a3-37ba-4595-9dc0-b354574eba07-istio-podinfo\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.472500 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.472471    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bced75a3-37ba-4595-9dc0-b354574eba07-istio-token\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.473013 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.472991    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbj75\" (UniqueName: \"kubernetes.io/projected/bced75a3-37ba-4595-9dc0-b354574eba07-kube-api-access-jbj75\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.506967 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.506943    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"
Oct 29 22:08:28.546924 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.546901    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-kube-api-access-stbkx\" (UniqueName: \"kubernetes.io/projected/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-kube-api-access-stbkx\") pod \"fb5e9803-65bc-4b86-815b-3f1bc323b3bb\" (UID: \"fb5e9803-65bc-4b86-815b-3f1bc323b3bb\") "
Oct 29 22:08:28.547120 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.547101    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-metrics-certs\") pod \"fb5e9803-65bc-4b86-815b-3f1bc323b3bb\" (UID: \"fb5e9803-65bc-4b86-815b-3f1bc323b3bb\") "
Oct 29 22:08:28.547230 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.547212    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-maas-api-tls\") pod \"fb5e9803-65bc-4b86-815b-3f1bc323b3bb\" (UID: \"fb5e9803-65bc-4b86-815b-3f1bc323b3bb\") "
Oct 29 22:08:28.547353 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.547334    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-openshift-service-ca\") pod \"fb5e9803-65bc-4b86-815b-3f1bc323b3bb\" (UID: \"fb5e9803-65bc-4b86-815b-3f1bc323b3bb\") "
Oct 29 22:08:28.547784 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.547755    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-openshift-service-ca" pod "fb5e9803-65bc-4b86-815b-3f1bc323b3bb" (UID: "fb5e9803-65bc-4b86-815b-3f1bc323b3bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:28.548407 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.548376    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.549623 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.549592    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-metrics-certs" pod "fb5e9803-65bc-4b86-815b-3f1bc323b3bb" (UID: "fb5e9803-65bc-4b86-815b-3f1bc323b3bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:08:28.549712 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.549630    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-kube-api-access-stbkx" pod "fb5e9803-65bc-4b86-815b-3f1bc323b3bb" (UID: "fb5e9803-65bc-4b86-815b-3f1bc323b3bb"). InnerVolumeSpecName "kube-api-access-stbkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:28.549813 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.549786    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-maas-api-tls" pod "fb5e9803-65bc-4b86-815b-3f1bc323b3bb" (UID: "fb5e9803-65bc-4b86-815b-3f1bc323b3bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:08:28.612219 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.612194    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:28.650088 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.650061    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.650088 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.650087    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:28.650243 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:28.650097    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stbkx\" (UniqueName: \"kubernetes.io/projected/fb5e9803-65bc-4b86-815b-3f1bc323b3bb-kube-api-access-stbkx\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:29.333378 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:29.332418    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"
Oct 29 22:08:29.404311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:29.325248    4962 generic.go:356] "Generic (PLEG): container finished" podID="fb5e9803-65bc-4b86-815b-3f1bc323b3bb" containerID="4b55130cebce2b54497dd37035b435da89cc38491b19adb1802ce1d15fbf644e" exitCode=0
Oct 29 22:08:29.599101 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:29.599018    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2f96700-2d4a-4b30-ada0-04cf70b7f6d2" path="/var/lib/kubelet/pods/c2f96700-2d4a-4b30-ada0-04cf70b7f6d2/volumes"
Oct 29 22:08:29.600704 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:29.600676    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6fe16e8-3928-4766-9daf-7a449c61233d" path="/var/lib/kubelet/pods/e6fe16e8-3928-4766-9daf-7a449c61233d/volumes"
Oct 29 22:08:29.602490 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:29.602460    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x" event={"ID":"fb5e9803-65bc-4b86-815b-3f1bc323b3bb","Type":"ContainerDied","Data":"4b55130cebce2b54497dd37035b435da89cc38491b19adb1802ce1d15fbf644e"}
Oct 29 22:08:29.602762 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:29.602709    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x" event={"ID":"fb5e9803-65bc-4b86-815b-3f1bc323b3bb","Type":"ContainerDied","Data":"81f88d01d297802c33d499734bd558e11431f6783cda253349b0a45eb8861be5"}
Oct 29 22:08:29.602915 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:29.602900    4962 scope.go:122] "RemoveContainer" containerID="4b55130cebce2b54497dd37035b435da89cc38491b19adb1802ce1d15fbf644e"
Oct 29 22:08:29.626113 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:29.626071    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:08:29.626352 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:29.626328    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:08:29.626498 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:29.626479    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:08:29.643171 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:29.642888    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"]
Oct 29 22:08:29.645159 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:29.645123    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"]
Oct 29 22:08:29.649997 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:29.649970    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-380eea-5f4984df5-cn25x"]
Oct 29 22:08:29.667778 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:29.667758    4962 scope.go:122] "RemoveContainer" containerID="4b55130cebce2b54497dd37035b435da89cc38491b19adb1802ce1d15fbf644e"
Oct 29 22:08:29.668973 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:08:29.668952    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b55130cebce2b54497dd37035b435da89cc38491b19adb1802ce1d15fbf644e\": container with ID starting with 4b55130cebce2b54497dd37035b435da89cc38491b19adb1802ce1d15fbf644e not found: ID does not exist" containerID="4b55130cebce2b54497dd37035b435da89cc38491b19adb1802ce1d15fbf644e"
Oct 29 22:08:29.669118 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:29.669092    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b55130cebce2b54497dd37035b435da89cc38491b19adb1802ce1d15fbf644e"} err="failed to get container status \"4b55130cebce2b54497dd37035b435da89cc38491b19adb1802ce1d15fbf644e\": rpc error: code = NotFound desc = could not find container \"4b55130cebce2b54497dd37035b435da89cc38491b19adb1802ce1d15fbf644e\": container with ID starting with 4b55130cebce2b54497dd37035b435da89cc38491b19adb1802ce1d15fbf644e not found: ID does not exist"
Oct 29 22:08:30.039032 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.039001    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4"
Oct 29 22:08:30.039386 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.039365    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf"
Oct 29 22:08:30.077032 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.076986    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn" podUID="28f4dbbf-6c33-49b8-a263-bf5956d62247" containerName="payload-processing" containerID="cri-o://aacc559172b4ba3274ec9e34f1497e801ca93cbd1c7dfd8c1d9d68f681c6fdf3" gracePeriod=25
Oct 29 22:08:30.109257 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.109225    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9" podUID="6dd2c6fb-be8c-48dd-9c74-e71293a09672" containerName="payload-pre-processing" containerID="cri-o://ed59f936975e2aecc54510e1ac9ccdb972139caf722be54fd6787258923e05ae" gracePeriod=25
Oct 29 22:08:30.409648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.409610    4962 generic.go:356] "Generic (PLEG): container finished" podID="6dd2c6fb-be8c-48dd-9c74-e71293a09672" containerID="ed59f936975e2aecc54510e1ac9ccdb972139caf722be54fd6787258923e05ae" exitCode=0
Oct 29 22:08:30.410032 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.409639    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9" event={"ID":"6dd2c6fb-be8c-48dd-9c74-e71293a09672","Type":"ContainerDied","Data":"ed59f936975e2aecc54510e1ac9ccdb972139caf722be54fd6787258923e05ae"}
Oct 29 22:08:30.412047 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.412015    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn" event={"ID":"bced75a3-37ba-4595-9dc0-b354574eba07","Type":"ContainerStarted","Data":"5ab3ad1e4ae32a29b221951ad5682083847f64fb401f6a51f2a83afd4d835689"}
Oct 29 22:08:30.412166 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.412066    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn" event={"ID":"bced75a3-37ba-4595-9dc0-b354574eba07","Type":"ContainerStarted","Data":"5b97bedba88180c27e5559e00f730a87a2958e1a744dbf9105a69b800bfb1798"}
Oct 29 22:08:30.416735 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.416710    4962 generic.go:356] "Generic (PLEG): container finished" podID="28f4dbbf-6c33-49b8-a263-bf5956d62247" containerID="aacc559172b4ba3274ec9e34f1497e801ca93cbd1c7dfd8c1d9d68f681c6fdf3" exitCode=0
Oct 29 22:08:30.416837 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.416823    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn" event={"ID":"28f4dbbf-6c33-49b8-a263-bf5956d62247","Type":"ContainerDied","Data":"aacc559172b4ba3274ec9e34f1497e801ca93cbd1c7dfd8c1d9d68f681c6fdf3"}
Oct 29 22:08:30.455117 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.455095    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn"
Oct 29 22:08:30.466059 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.466016    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn" podStartSLOduration=2.466002724 podStartE2EDuration="2.466002724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:30.464155774 +0000 UTC m=+1331.537112763" watchObservedRunningTime="2026-10-29 22:08:30.466002724 +0000 UTC m=+1331.538959687"
Oct 29 22:08:30.526265 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.526232    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28f4dbbf-6c33-49b8-a263-bf5956d62247-kube-api-access-94fff\" (UniqueName: \"kubernetes.io/projected/28f4dbbf-6c33-49b8-a263-bf5956d62247-kube-api-access-94fff\") pod \"28f4dbbf-6c33-49b8-a263-bf5956d62247\" (UID: \"28f4dbbf-6c33-49b8-a263-bf5956d62247\") "
Oct 29 22:08:30.526613 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.526590    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28f4dbbf-6c33-49b8-a263-bf5956d62247-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28f4dbbf-6c33-49b8-a263-bf5956d62247-plugins-config-volume\") pod \"28f4dbbf-6c33-49b8-a263-bf5956d62247\" (UID: \"28f4dbbf-6c33-49b8-a263-bf5956d62247\") "
Oct 29 22:08:30.527928 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.527891    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28f4dbbf-6c33-49b8-a263-bf5956d62247-plugins-config-volume" pod "28f4dbbf-6c33-49b8-a263-bf5956d62247" (UID: "28f4dbbf-6c33-49b8-a263-bf5956d62247"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:30.529117 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.529093    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f4dbbf-6c33-49b8-a263-bf5956d62247-kube-api-access-94fff" pod "28f4dbbf-6c33-49b8-a263-bf5956d62247" (UID: "28f4dbbf-6c33-49b8-a263-bf5956d62247"). InnerVolumeSpecName "kube-api-access-94fff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:30.537340 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.535980    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9"
Oct 29 22:08:30.709307 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.622465    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:30.741837 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.741809    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:30.747240 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.665405    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94fff\" (UniqueName: \"kubernetes.io/projected/28f4dbbf-6c33-49b8-a263-bf5956d62247-kube-api-access-94fff\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:30.751411 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.751385    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28f4dbbf-6c33-49b8-a263-bf5956d62247-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:30.853109 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.853076    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dd2c6fb-be8c-48dd-9c74-e71293a09672-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd2c6fb-be8c-48dd-9c74-e71293a09672-plugins-config-volume\") pod \"6dd2c6fb-be8c-48dd-9c74-e71293a09672\" (UID: \"6dd2c6fb-be8c-48dd-9c74-e71293a09672\") "
Oct 29 22:08:30.853355 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.853120    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dd2c6fb-be8c-48dd-9c74-e71293a09672-kube-api-access-rqd58\" (UniqueName: \"kubernetes.io/projected/6dd2c6fb-be8c-48dd-9c74-e71293a09672-kube-api-access-rqd58\") pod \"6dd2c6fb-be8c-48dd-9c74-e71293a09672\" (UID: \"6dd2c6fb-be8c-48dd-9c74-e71293a09672\") "
Oct 29 22:08:30.853545 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.853520    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd2c6fb-be8c-48dd-9c74-e71293a09672-plugins-config-volume" pod "6dd2c6fb-be8c-48dd-9c74-e71293a09672" (UID: "6dd2c6fb-be8c-48dd-9c74-e71293a09672"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:30.853816 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.853798    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd2c6fb-be8c-48dd-9c74-e71293a09672-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:30.855300 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.855268    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd2c6fb-be8c-48dd-9c74-e71293a09672-kube-api-access-rqd58" pod "6dd2c6fb-be8c-48dd-9c74-e71293a09672" (UID: "6dd2c6fb-be8c-48dd-9c74-e71293a09672"). InnerVolumeSpecName "kube-api-access-rqd58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:30.955378 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:30.955350    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqd58\" (UniqueName: \"kubernetes.io/projected/6dd2c6fb-be8c-48dd-9c74-e71293a09672-kube-api-access-rqd58\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:31.061142 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:31.061076    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb5e9803-65bc-4b86-815b-3f1bc323b3bb" path="/var/lib/kubelet/pods/fb5e9803-65bc-4b86-815b-3f1bc323b3bb/volumes"
Oct 29 22:08:31.428967 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:31.428939    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn"
Oct 29 22:08:31.429411 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:31.428939    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn" event={"ID":"28f4dbbf-6c33-49b8-a263-bf5956d62247","Type":"ContainerDied","Data":"03e5abeea91903834f90b2d713d6b5f35ebfe26e603628f581e85d57ede754cc"}
Oct 29 22:08:31.429411 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:31.429132    4962 scope.go:122] "RemoveContainer" containerID="aacc559172b4ba3274ec9e34f1497e801ca93cbd1c7dfd8c1d9d68f681c6fdf3"
Oct 29 22:08:31.430880 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:31.430839    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9"
Oct 29 22:08:31.430880 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:31.430861    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9" event={"ID":"6dd2c6fb-be8c-48dd-9c74-e71293a09672","Type":"ContainerDied","Data":"09327c8df7373b4418967f6848b77031179dc0962b6322713cc41d0b803df10f"}
Oct 29 22:08:31.431303 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:31.431263    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:31.432204 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:31.432182    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:08:31.440391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:31.440370    4962 scope.go:122] "RemoveContainer" containerID="ed59f936975e2aecc54510e1ac9ccdb972139caf722be54fd6787258923e05ae"
Oct 29 22:08:31.495443 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:31.495422    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9"]
Oct 29 22:08:31.501863 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:31.501843    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-df42c7-84466d8cd7-j29r9"]
Oct 29 22:08:31.595863 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:31.595830    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn"]
Oct 29 22:08:31.603177 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:31.603145    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-df42c7-d98767f94-29cmn"]
Oct 29 22:08:32.528377 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.528348    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"]
Oct 29 22:08:32.529384 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.529359    4962 cpu_manager.go:414] "containerMap: removing container" podUID="6dd2c6fb-be8c-48dd-9c74-e71293a09672" containerName="payload-pre-processing"
Oct 29 22:08:32.529384 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.529382    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd2c6fb-be8c-48dd-9c74-e71293a09672" containerName="payload-pre-processing"
Oct 29 22:08:32.529559 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.529447    4962 cpu_manager.go:414] "containerMap: removing container" podUID="28f4dbbf-6c33-49b8-a263-bf5956d62247" containerName="payload-processing"
Oct 29 22:08:32.529559 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.529456    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f4dbbf-6c33-49b8-a263-bf5956d62247" containerName="payload-processing"
Oct 29 22:08:32.529559 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.529479    4962 cpu_manager.go:414] "containerMap: removing container" podUID="fb5e9803-65bc-4b86-815b-3f1bc323b3bb" containerName="maas-api"
Oct 29 22:08:32.529559 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.529486    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb5e9803-65bc-4b86-815b-3f1bc323b3bb" containerName="maas-api"
Oct 29 22:08:32.529754 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.529674    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f4dbbf-6c33-49b8-a263-bf5956d62247" containerName="payload-processing"
Oct 29 22:08:32.529754 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.529693    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb5e9803-65bc-4b86-815b-3f1bc323b3bb" containerName="maas-api"
Oct 29 22:08:32.529754 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.529709    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd2c6fb-be8c-48dd-9c74-e71293a09672" containerName="payload-pre-processing"
Oct 29 22:08:32.535149 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.535127    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.590274 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.590237    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"]
Oct 29 22:08:32.599483 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.599451    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.599793 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.599498    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.599793 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.599701    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.599793 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.599780    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.599957 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.599834    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.599957 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.599851    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.599957 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.599869    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.599957 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.599941    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m86fx\" (UniqueName: \"kubernetes.io/projected/cbd6251b-df2f-4739-ae5d-8adbd277e52c-kube-api-access-m86fx\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.600072 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.600026    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.701157 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.701126    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.701329 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.701203    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.701329 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.701263    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.701329 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.701280    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.701329 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.701315    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.701580 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.701400    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m86fx\" (UniqueName: \"kubernetes.io/projected/cbd6251b-df2f-4739-ae5d-8adbd277e52c-kube-api-access-m86fx\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.701580 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.701547    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.701695 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.701609    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.701695 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.701605    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-workload-certs\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.701695 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.701633    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.701880 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.701706    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-credential-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.701956 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.701929    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-data\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.702007 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.701965    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-workload-socket\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.702007 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.701990    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istiod-ca-cert\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.703855 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.703835    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-envoy\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.703960 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.703920    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-podinfo\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.738434 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.738413    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-token\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.738582 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.738560    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m86fx\" (UniqueName: \"kubernetes.io/projected/cbd6251b-df2f-4739-ae5d-8adbd277e52c-kube-api-access-m86fx\") pod \"e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") " pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:32.851431 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:32.851358    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:33.062208 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:33.062174    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28f4dbbf-6c33-49b8-a263-bf5956d62247" path="/var/lib/kubelet/pods/28f4dbbf-6c33-49b8-a263-bf5956d62247/volumes"
Oct 29 22:08:33.062707 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:33.062691    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dd2c6fb-be8c-48dd-9c74-e71293a09672" path="/var/lib/kubelet/pods/6dd2c6fb-be8c-48dd-9c74-e71293a09672/volumes"
Oct 29 22:08:33.200298 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:08:33.200259    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbd6251b_df2f_4739_ae5d_8adbd277e52c.slice/crio-ffcb58f56f475baa91bbaa964f4c54814b644a456b8f9e8141711d696aec6674 WatchSource:0}: Error finding container ffcb58f56f475baa91bbaa964f4c54814b644a456b8f9e8141711d696aec6674: Status 404 returned error can't find the container with id ffcb58f56f475baa91bbaa964f4c54814b644a456b8f9e8141711d696aec6674
Oct 29 22:08:33.202978 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:33.202954    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"]
Oct 29 22:08:33.203216 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:33.203187    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:08:33.203263 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:33.203251    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:08:33.203306 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:33.203280    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:08:33.451794 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:33.451712    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b" event={"ID":"cbd6251b-df2f-4739-ae5d-8adbd277e52c","Type":"ContainerStarted","Data":"dfc0898b2a96c3f21ca0edfc46c187b4a86426ecfa4fd8e31e709aab8893bdea"}
Oct 29 22:08:33.451794 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:33.451772    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b" event={"ID":"cbd6251b-df2f-4739-ae5d-8adbd277e52c","Type":"ContainerStarted","Data":"ffcb58f56f475baa91bbaa964f4c54814b644a456b8f9e8141711d696aec6674"}
Oct 29 22:08:33.510613 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:33.510565    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b" podStartSLOduration=1.510549693 podStartE2EDuration="1.510549693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:33.507887483 +0000 UTC m=+1334.580844447" watchObservedRunningTime="2026-10-29 22:08:33.510549693 +0000 UTC m=+1334.583506656"
Oct 29 22:08:33.851610 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:33.851534    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:34.857178 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:34.857147    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:35.066046 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:35.066014    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b"]
Oct 29 22:08:35.070839 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:35.070820    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b"
Oct 29 22:08:35.090873 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:35.090844    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a9e67f-dockercfg-48s4s\""
Oct 29 22:08:35.090999 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:35.090901    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a9e67f\""
Oct 29 22:08:35.095392 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:35.095372    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b"]
Oct 29 22:08:35.284149 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:35.240934    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/528077a0-2c34-4ff3-b229-3af6a3cebb16-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b\" (UID: \"528077a0-2c34-4ff3-b229-3af6a3cebb16\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b"
Oct 29 22:08:35.284149 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:35.241075    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6cgj\" (UniqueName: \"kubernetes.io/projected/528077a0-2c34-4ff3-b229-3af6a3cebb16-kube-api-access-p6cgj\") pod \"payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b\" (UID: \"528077a0-2c34-4ff3-b229-3af6a3cebb16\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b"
Oct 29 22:08:35.402311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:35.377360    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9"]
Oct 29 22:08:35.429680 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:35.429641    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9"
Oct 29 22:08:35.506303 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:35.506247    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/528077a0-2c34-4ff3-b229-3af6a3cebb16-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b\" (UID: \"528077a0-2c34-4ff3-b229-3af6a3cebb16\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b"
Oct 29 22:08:35.517521 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:35.517476    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6cgj\" (UniqueName: \"kubernetes.io/projected/528077a0-2c34-4ff3-b229-3af6a3cebb16-kube-api-access-p6cgj\") pod \"payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b\" (UID: \"528077a0-2c34-4ff3-b229-3af6a3cebb16\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b"
Oct 29 22:08:35.521035 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:35.521009    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/528077a0-2c34-4ff3-b229-3af6a3cebb16-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b\" (UID: \"528077a0-2c34-4ff3-b229-3af6a3cebb16\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b"
Oct 29 22:08:35.605653 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:35.602445    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6cgj\" (UniqueName: \"kubernetes.io/projected/528077a0-2c34-4ff3-b229-3af6a3cebb16-kube-api-access-p6cgj\") pod \"payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b\" (UID: \"528077a0-2c34-4ff3-b229-3af6a3cebb16\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b"
Oct 29 22:08:35.621529 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:35.620660    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9d173f-c37f-4675-9ffe-9ca09e25794c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9\" (UID: \"3e9d173f-c37f-4675-9ffe-9ca09e25794c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9"
Oct 29 22:08:35.621529 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:35.620862    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fd7x\" (UniqueName: \"kubernetes.io/projected/3e9d173f-c37f-4675-9ffe-9ca09e25794c-kube-api-access-5fd7x\") pod \"payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9\" (UID: \"3e9d173f-c37f-4675-9ffe-9ca09e25794c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9"
Oct 29 22:08:35.652365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:35.652332    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:35.653800 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:35.653766    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:08:35.673406 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:35.673370    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9"]
Oct 29 22:08:35.687056 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:35.687000    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b"
Oct 29 22:08:35.727955 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:35.723463    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9d173f-c37f-4675-9ffe-9ca09e25794c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9\" (UID: \"3e9d173f-c37f-4675-9ffe-9ca09e25794c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9"
Oct 29 22:08:35.727955 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:35.723547    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fd7x\" (UniqueName: \"kubernetes.io/projected/3e9d173f-c37f-4675-9ffe-9ca09e25794c-kube-api-access-5fd7x\") pod \"payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9\" (UID: \"3e9d173f-c37f-4675-9ffe-9ca09e25794c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9"
Oct 29 22:08:35.727955 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:35.725109    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9d173f-c37f-4675-9ffe-9ca09e25794c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9\" (UID: \"3e9d173f-c37f-4675-9ffe-9ca09e25794c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9"
Oct 29 22:08:35.749711 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:35.749647    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fd7x\" (UniqueName: \"kubernetes.io/projected/3e9d173f-c37f-4675-9ffe-9ca09e25794c-kube-api-access-5fd7x\") pod \"payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9\" (UID: \"3e9d173f-c37f-4675-9ffe-9ca09e25794c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9"
Oct 29 22:08:35.958639 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:35.958607    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9"
Oct 29 22:08:36.023335 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:36.022425    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b"]
Oct 29 22:08:36.023335 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:08:36.023208    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod528077a0_2c34_4ff3_b229_3af6a3cebb16.slice/crio-bd13b00614ad4c8eec324d34cfd2deab3af17a96e9abbe66650bbfbd26a86517 WatchSource:0}: Error finding container bd13b00614ad4c8eec324d34cfd2deab3af17a96e9abbe66650bbfbd26a86517: Status 404 returned error can't find the container with id bd13b00614ad4c8eec324d34cfd2deab3af17a96e9abbe66650bbfbd26a86517
Oct 29 22:08:36.288783 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:08:36.288755    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9d173f_c37f_4675_9ffe_9ca09e25794c.slice/crio-53fe6fbcd77466181dd48af8249b0fc249aa86d247fa04022f84d23ae45efc61 WatchSource:0}: Error finding container 53fe6fbcd77466181dd48af8249b0fc249aa86d247fa04022f84d23ae45efc61: Status 404 returned error can't find the container with id 53fe6fbcd77466181dd48af8249b0fc249aa86d247fa04022f84d23ae45efc61
Oct 29 22:08:36.290600 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:36.290578    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9"]
Oct 29 22:08:36.668152 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:36.667842    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b" event={"ID":"528077a0-2c34-4ff3-b229-3af6a3cebb16","Type":"ContainerStarted","Data":"1f72ba6c609487567e1713ad1b605b67e272531b867e565c4abd03bf0b76ddd5"}
Oct 29 22:08:36.668152 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:36.667935    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b" event={"ID":"528077a0-2c34-4ff3-b229-3af6a3cebb16","Type":"ContainerStarted","Data":"bd13b00614ad4c8eec324d34cfd2deab3af17a96e9abbe66650bbfbd26a86517"}
Oct 29 22:08:36.668152 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:36.668113    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b"
Oct 29 22:08:36.669839 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:36.669811    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9" event={"ID":"3e9d173f-c37f-4675-9ffe-9ca09e25794c","Type":"ContainerStarted","Data":"ae5cea13586587b5e228bdfbcdd1a3f0ac685c42ac83c5ef467b748a742bd1c1"}
Oct 29 22:08:36.669943 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:36.669861    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9" event={"ID":"3e9d173f-c37f-4675-9ffe-9ca09e25794c","Type":"ContainerStarted","Data":"53fe6fbcd77466181dd48af8249b0fc249aa86d247fa04022f84d23ae45efc61"}
Oct 29 22:08:36.669943 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:36.669891    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9"
Oct 29 22:08:36.713152 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:36.713113    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b" podStartSLOduration=1.713101598 podStartE2EDuration="1.713101598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:36.7084496 +0000 UTC m=+1337.781406565" watchObservedRunningTime="2026-10-29 22:08:36.713101598 +0000 UTC m=+1337.786058604"
Oct 29 22:08:36.755403 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:36.755354    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9" podStartSLOduration=1.7553414250000001 podStartE2EDuration="1.755341425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:36.751994861 +0000 UTC m=+1337.824951824" watchObservedRunningTime="2026-10-29 22:08:36.755341425 +0000 UTC m=+1337.828298390"
Oct 29 22:08:39.564204 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:39.564169    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"]
Oct 29 22:08:39.564738 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:39.564470    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg" podUID="9b3220a5-a632-40bd-b9ad-668bb2b5b237" containerName="istio-proxy" containerID="cri-o://35a4dc9d31753702f3d8d75b201fa29ab3660c53a9bac13fbab6ee1c41d9a70e" gracePeriod=30
Oct 29 22:08:39.582888 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:39.582856    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"]
Oct 29 22:08:39.585204 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:39.585167    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk" podUID="1106f45f-cdca-428f-8a2e-dc402d199ae8" containerName="istio-proxy" containerID="cri-o://2c1e96e174e66a2eab4f5e6502425730eae7015634f90fbf22e40dabb04b7ff0" gracePeriod=30
Oct 29 22:08:39.595585 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:39.595561    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"]
Oct 29 22:08:39.595801 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:39.595780    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25" podUID="a93430b5-4141-4183-a6f4-11297e0384c0" containerName="istio-proxy" containerID="cri-o://1782df9d6fc7b77793e388f65492408da83206592cdea0cc34ebfa08d465e256" gracePeriod=30
Oct 29 22:08:40.042269 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:08:40.037487    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:08:44.951235 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:44.951202    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:08:44.964918 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:44.964889    4962 generic.go:356] "Generic (PLEG): container finished" podID="a93430b5-4141-4183-a6f4-11297e0384c0" containerID="1782df9d6fc7b77793e388f65492408da83206592cdea0cc34ebfa08d465e256" exitCode=0
Oct 29 22:08:44.965061 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:44.965013    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25" event={"ID":"a93430b5-4141-4183-a6f4-11297e0384c0","Type":"ContainerDied","Data":"1782df9d6fc7b77793e388f65492408da83206592cdea0cc34ebfa08d465e256"}
Oct 29 22:08:44.966985 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:44.966960    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:08:44.979194 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:44.979167    4962 generic.go:356] "Generic (PLEG): container finished" podID="1106f45f-cdca-428f-8a2e-dc402d199ae8" containerID="2c1e96e174e66a2eab4f5e6502425730eae7015634f90fbf22e40dabb04b7ff0" exitCode=0
Oct 29 22:08:44.979616 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:44.979592    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk" event={"ID":"1106f45f-cdca-428f-8a2e-dc402d199ae8","Type":"ContainerDied","Data":"2c1e96e174e66a2eab4f5e6502425730eae7015634f90fbf22e40dabb04b7ff0"}
Oct 29 22:08:44.979837 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:44.979815    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk" event={"ID":"1106f45f-cdca-428f-8a2e-dc402d199ae8","Type":"ContainerDied","Data":"91e07c2abdc914a1a77dfc96e2f9a6add574d2383b6b75455bbe51a3f9e16f3b"}
Oct 29 22:08:44.979926 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:44.979868    4962 scope.go:122] "RemoveContainer" containerID="2c1e96e174e66a2eab4f5e6502425730eae7015634f90fbf22e40dabb04b7ff0"
Oct 29 22:08:44.980091 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:44.980076    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"
Oct 29 22:08:44.988929 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:44.984819    4962 generic.go:356] "Generic (PLEG): container finished" podID="9b3220a5-a632-40bd-b9ad-668bb2b5b237" containerID="35a4dc9d31753702f3d8d75b201fa29ab3660c53a9bac13fbab6ee1c41d9a70e" exitCode=0
Oct 29 22:08:44.988929 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:44.985081    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg" event={"ID":"9b3220a5-a632-40bd-b9ad-668bb2b5b237","Type":"ContainerDied","Data":"35a4dc9d31753702f3d8d75b201fa29ab3660c53a9bac13fbab6ee1c41d9a70e"}
Oct 29 22:08:44.988929 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:44.985144    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg" event={"ID":"9b3220a5-a632-40bd-b9ad-668bb2b5b237","Type":"ContainerDied","Data":"ef7be4397f63562a3ca111d5d1ca59dcd527bbd7064b251d48f5aa754bd9884d"}
Oct 29 22:08:44.988929 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:44.985258    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"
Oct 29 22:08:45.005678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.005654    4962 scope.go:122] "RemoveContainer" containerID="2c1e96e174e66a2eab4f5e6502425730eae7015634f90fbf22e40dabb04b7ff0"
Oct 29 22:08:45.006030 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:08:45.005997    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c1e96e174e66a2eab4f5e6502425730eae7015634f90fbf22e40dabb04b7ff0\": container with ID starting with 2c1e96e174e66a2eab4f5e6502425730eae7015634f90fbf22e40dabb04b7ff0 not found: ID does not exist" containerID="2c1e96e174e66a2eab4f5e6502425730eae7015634f90fbf22e40dabb04b7ff0"
Oct 29 22:08:45.006120 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.006041    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c1e96e174e66a2eab4f5e6502425730eae7015634f90fbf22e40dabb04b7ff0"} err="failed to get container status \"2c1e96e174e66a2eab4f5e6502425730eae7015634f90fbf22e40dabb04b7ff0\": rpc error: code = NotFound desc = could not find container \"2c1e96e174e66a2eab4f5e6502425730eae7015634f90fbf22e40dabb04b7ff0\": container with ID starting with 2c1e96e174e66a2eab4f5e6502425730eae7015634f90fbf22e40dabb04b7ff0 not found: ID does not exist"
Oct 29 22:08:45.006120 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.006066    4962 scope.go:122] "RemoveContainer" containerID="35a4dc9d31753702f3d8d75b201fa29ab3660c53a9bac13fbab6ee1c41d9a70e"
Oct 29 22:08:45.012509 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.012489    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:08:45.025252 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.025235    4962 scope.go:122] "RemoveContainer" containerID="35a4dc9d31753702f3d8d75b201fa29ab3660c53a9bac13fbab6ee1c41d9a70e"
Oct 29 22:08:45.025574 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:08:45.025536    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a4dc9d31753702f3d8d75b201fa29ab3660c53a9bac13fbab6ee1c41d9a70e\": container with ID starting with 35a4dc9d31753702f3d8d75b201fa29ab3660c53a9bac13fbab6ee1c41d9a70e not found: ID does not exist" containerID="35a4dc9d31753702f3d8d75b201fa29ab3660c53a9bac13fbab6ee1c41d9a70e"
Oct 29 22:08:45.025688 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.025569    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a4dc9d31753702f3d8d75b201fa29ab3660c53a9bac13fbab6ee1c41d9a70e"} err="failed to get container status \"35a4dc9d31753702f3d8d75b201fa29ab3660c53a9bac13fbab6ee1c41d9a70e\": rpc error: code = NotFound desc = could not find container \"35a4dc9d31753702f3d8d75b201fa29ab3660c53a9bac13fbab6ee1c41d9a70e\": container with ID starting with 35a4dc9d31753702f3d8d75b201fa29ab3660c53a9bac13fbab6ee1c41d9a70e not found: ID does not exist"
Oct 29 22:08:45.043510 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.043486    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-istio-data\") pod \"a93430b5-4141-4183-a6f4-11297e0384c0\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") "
Oct 29 22:08:45.043618 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.043519    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-istio-envoy\") pod \"a93430b5-4141-4183-a6f4-11297e0384c0\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") "
Oct 29 22:08:45.043618 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.043538    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-workload-certs\") pod \"1106f45f-cdca-428f-8a2e-dc402d199ae8\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") "
Oct 29 22:08:45.043618 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.043558    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a93430b5-4141-4183-a6f4-11297e0384c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a93430b5-4141-4183-a6f4-11297e0384c0-istio-podinfo\") pod \"a93430b5-4141-4183-a6f4-11297e0384c0\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") "
Oct 29 22:08:45.043847 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.043630    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istiod-ca-cert\") pod \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") "
Oct 29 22:08:45.043847 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.043657    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a93430b5-4141-4183-a6f4-11297e0384c0-kube-api-access-hrp49\" (UniqueName: \"kubernetes.io/projected/a93430b5-4141-4183-a6f4-11297e0384c0-kube-api-access-hrp49\") pod \"a93430b5-4141-4183-a6f4-11297e0384c0\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") "
Oct 29 22:08:45.043847 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.043685    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-data\") pod \"1106f45f-cdca-428f-8a2e-dc402d199ae8\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") "
Oct 29 22:08:45.043847 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.043704    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-credential-socket\") pod \"a93430b5-4141-4183-a6f4-11297e0384c0\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") "
Oct 29 22:08:45.043847 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.043741    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a93430b5-4141-4183-a6f4-11297e0384c0-istio-token\" (UniqueName: \"kubernetes.io/projected/a93430b5-4141-4183-a6f4-11297e0384c0-istio-token\") pod \"a93430b5-4141-4183-a6f4-11297e0384c0\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") "
Oct 29 22:08:45.043847 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.043815    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-workload-certs\") pod \"a93430b5-4141-4183-a6f4-11297e0384c0\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") "
Oct 29 22:08:45.044153 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.043858    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-credential-socket\") pod \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") "
Oct 29 22:08:45.044153 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.043936    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-envoy\") pod \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") "
Oct 29 22:08:45.044153 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.044007    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-workload-certs\") pod \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") "
Oct 29 22:08:45.044153 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.044040    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-data\") pod \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") "
Oct 29 22:08:45.044153 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.044067    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-token\" (UniqueName: \"kubernetes.io/projected/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-token\") pod \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") "
Oct 29 22:08:45.044153 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.044141    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1106f45f-cdca-428f-8a2e-dc402d199ae8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1106f45f-cdca-428f-8a2e-dc402d199ae8-istiod-ca-cert\") pod \"1106f45f-cdca-428f-8a2e-dc402d199ae8\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") "
Oct 29 22:08:45.044502 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.044231    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-workload-socket\") pod \"1106f45f-cdca-428f-8a2e-dc402d199ae8\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") "
Oct 29 22:08:45.044502 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.044270    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a93430b5-4141-4183-a6f4-11297e0384c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a93430b5-4141-4183-a6f4-11297e0384c0-istiod-ca-cert\") pod \"a93430b5-4141-4183-a6f4-11297e0384c0\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") "
Oct 29 22:08:45.044502 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.044338    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-envoy\") pod \"1106f45f-cdca-428f-8a2e-dc402d199ae8\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") "
Oct 29 22:08:45.044502 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.044421    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1106f45f-cdca-428f-8a2e-dc402d199ae8-kube-api-access-2zzdx\" (UniqueName: \"kubernetes.io/projected/1106f45f-cdca-428f-8a2e-dc402d199ae8-kube-api-access-2zzdx\") pod \"1106f45f-cdca-428f-8a2e-dc402d199ae8\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") "
Oct 29 22:08:45.044502 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.044491    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-workload-socket\") pod \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") "
Oct 29 22:08:45.044759 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.044540    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-credential-socket\") pod \"1106f45f-cdca-428f-8a2e-dc402d199ae8\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") "
Oct 29 22:08:45.044759 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.044567    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-workload-socket\") pod \"a93430b5-4141-4183-a6f4-11297e0384c0\" (UID: \"a93430b5-4141-4183-a6f4-11297e0384c0\") "
Oct 29 22:08:45.044759 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.044611    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-token\" (UniqueName: \"kubernetes.io/projected/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-token\") pod \"1106f45f-cdca-428f-8a2e-dc402d199ae8\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") "
Oct 29 22:08:45.044759 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.044652    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b3220a5-a632-40bd-b9ad-668bb2b5b237-kube-api-access-bsgg2\" (UniqueName: \"kubernetes.io/projected/9b3220a5-a632-40bd-b9ad-668bb2b5b237-kube-api-access-bsgg2\") pod \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") "
Oct 29 22:08:45.044759 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.044680    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-podinfo\") pod \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\" (UID: \"9b3220a5-a632-40bd-b9ad-668bb2b5b237\") "
Oct 29 22:08:45.044759 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.044721    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-podinfo\") pod \"1106f45f-cdca-428f-8a2e-dc402d199ae8\" (UID: \"1106f45f-cdca-428f-8a2e-dc402d199ae8\") "
Oct 29 22:08:45.048612 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.048585    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-podinfo" pod "1106f45f-cdca-428f-8a2e-dc402d199ae8" (UID: "1106f45f-cdca-428f-8a2e-dc402d199ae8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:08:45.048766 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.048751    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-istio-data" pod "a93430b5-4141-4183-a6f4-11297e0384c0" (UID: "a93430b5-4141-4183-a6f4-11297e0384c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.050586 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.050562    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-istio-envoy" pod "a93430b5-4141-4183-a6f4-11297e0384c0" (UID: "a93430b5-4141-4183-a6f4-11297e0384c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.050956 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.050926    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-workload-certs" pod "1106f45f-cdca-428f-8a2e-dc402d199ae8" (UID: "1106f45f-cdca-428f-8a2e-dc402d199ae8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.051816 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.051790    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istiod-ca-cert" pod "9b3220a5-a632-40bd-b9ad-668bb2b5b237" (UID: "9b3220a5-a632-40bd-b9ad-668bb2b5b237"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:45.052682 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.052531    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-data" pod "1106f45f-cdca-428f-8a2e-dc402d199ae8" (UID: "1106f45f-cdca-428f-8a2e-dc402d199ae8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.053125 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.053094    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-workload-certs" pod "a93430b5-4141-4183-a6f4-11297e0384c0" (UID: "a93430b5-4141-4183-a6f4-11297e0384c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.053125 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.053113    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-workload-socket" pod "a93430b5-4141-4183-a6f4-11297e0384c0" (UID: "a93430b5-4141-4183-a6f4-11297e0384c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.053297 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.053212    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-credential-socket" pod "1106f45f-cdca-428f-8a2e-dc402d199ae8" (UID: "1106f45f-cdca-428f-8a2e-dc402d199ae8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.053609 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.053579    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-credential-socket" pod "9b3220a5-a632-40bd-b9ad-668bb2b5b237" (UID: "9b3220a5-a632-40bd-b9ad-668bb2b5b237"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.053609 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.053600    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a93430b5-4141-4183-a6f4-11297e0384c0-istio-podinfo" pod "a93430b5-4141-4183-a6f4-11297e0384c0" (UID: "a93430b5-4141-4183-a6f4-11297e0384c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:08:45.053781 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.053587    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-workload-socket" pod "1106f45f-cdca-428f-8a2e-dc402d199ae8" (UID: "1106f45f-cdca-428f-8a2e-dc402d199ae8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.053781 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.053676    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-credential-socket" pod "a93430b5-4141-4183-a6f4-11297e0384c0" (UID: "a93430b5-4141-4183-a6f4-11297e0384c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.053781 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.053717    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-workload-certs" pod "9b3220a5-a632-40bd-b9ad-668bb2b5b237" (UID: "9b3220a5-a632-40bd-b9ad-668bb2b5b237"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.054151 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.054033    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-data" pod "9b3220a5-a632-40bd-b9ad-668bb2b5b237" (UID: "9b3220a5-a632-40bd-b9ad-668bb2b5b237"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.054597 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.054528    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a93430b5-4141-4183-a6f4-11297e0384c0-istiod-ca-cert" pod "a93430b5-4141-4183-a6f4-11297e0384c0" (UID: "a93430b5-4141-4183-a6f4-11297e0384c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:45.056614 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.056582    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93430b5-4141-4183-a6f4-11297e0384c0-kube-api-access-hrp49" pod "a93430b5-4141-4183-a6f4-11297e0384c0" (UID: "a93430b5-4141-4183-a6f4-11297e0384c0"). InnerVolumeSpecName "kube-api-access-hrp49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:45.056696 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.056650    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-envoy" pod "9b3220a5-a632-40bd-b9ad-668bb2b5b237" (UID: "9b3220a5-a632-40bd-b9ad-668bb2b5b237"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.057016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.056995    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1106f45f-cdca-428f-8a2e-dc402d199ae8-istiod-ca-cert" pod "1106f45f-cdca-428f-8a2e-dc402d199ae8" (UID: "1106f45f-cdca-428f-8a2e-dc402d199ae8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:45.057257 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.057238    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-workload-socket" pod "9b3220a5-a632-40bd-b9ad-668bb2b5b237" (UID: "9b3220a5-a632-40bd-b9ad-668bb2b5b237"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.060578 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.060540    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1106f45f-cdca-428f-8a2e-dc402d199ae8-kube-api-access-2zzdx" pod "1106f45f-cdca-428f-8a2e-dc402d199ae8" (UID: "1106f45f-cdca-428f-8a2e-dc402d199ae8"). InnerVolumeSpecName "kube-api-access-2zzdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:45.067763 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.067729    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b3220a5-a632-40bd-b9ad-668bb2b5b237-kube-api-access-bsgg2" pod "9b3220a5-a632-40bd-b9ad-668bb2b5b237" (UID: "9b3220a5-a632-40bd-b9ad-668bb2b5b237"). InnerVolumeSpecName "kube-api-access-bsgg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:45.070728 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.070704    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93430b5-4141-4183-a6f4-11297e0384c0-istio-token" pod "a93430b5-4141-4183-a6f4-11297e0384c0" (UID: "a93430b5-4141-4183-a6f4-11297e0384c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:45.072528 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.072439    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-token" pod "9b3220a5-a632-40bd-b9ad-668bb2b5b237" (UID: "9b3220a5-a632-40bd-b9ad-668bb2b5b237"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:45.072528 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.072445    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-podinfo" pod "9b3220a5-a632-40bd-b9ad-668bb2b5b237" (UID: "9b3220a5-a632-40bd-b9ad-668bb2b5b237"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:08:45.072528 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.072473    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-token" pod "1106f45f-cdca-428f-8a2e-dc402d199ae8" (UID: "1106f45f-cdca-428f-8a2e-dc402d199ae8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:45.072749 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.072548    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-envoy" pod "1106f45f-cdca-428f-8a2e-dc402d199ae8" (UID: "1106f45f-cdca-428f-8a2e-dc402d199ae8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:08:45.149373 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149280    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.149572 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149551    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrp49\" (UniqueName: \"kubernetes.io/projected/a93430b5-4141-4183-a6f4-11297e0384c0-kube-api-access-hrp49\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.149669 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149574    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.149669 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149589    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.149669 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149601    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a93430b5-4141-4183-a6f4-11297e0384c0-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.149669 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149615    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.149669 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149627    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.149669 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149640    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.149669 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149652    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.149669 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149664    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.150016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149676    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.150016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149688    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1106f45f-cdca-428f-8a2e-dc402d199ae8-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.150016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149701    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.150016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149713    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a93430b5-4141-4183-a6f4-11297e0384c0-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.150016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149726    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.150016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149740    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zzdx\" (UniqueName: \"kubernetes.io/projected/1106f45f-cdca-428f-8a2e-dc402d199ae8-kube-api-access-2zzdx\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.150016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149753    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3220a5-a632-40bd-b9ad-668bb2b5b237-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.150016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149768    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.150016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149783    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.150016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149797    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.150016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149811    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsgg2\" (UniqueName: \"kubernetes.io/projected/9b3220a5-a632-40bd-b9ad-668bb2b5b237-kube-api-access-bsgg2\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.150016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149826    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b3220a5-a632-40bd-b9ad-668bb2b5b237-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.150016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149839    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1106f45f-cdca-428f-8a2e-dc402d199ae8-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.150016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149852    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.150016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149868    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a93430b5-4141-4183-a6f4-11297e0384c0-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.150016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149880    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1106f45f-cdca-428f-8a2e-dc402d199ae8-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.150016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.149894    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a93430b5-4141-4183-a6f4-11297e0384c0-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:45.324564 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.324530    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"]
Oct 29 22:08:45.337098 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.337071    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-wdqjk"]
Oct 29 22:08:45.371681 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.371655    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"]
Oct 29 22:08:45.381340 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.381316    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-52rtg"]
Oct 29 22:08:45.993994 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.993969    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"
Oct 29 22:08:45.994481 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.994000    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25" event={"ID":"a93430b5-4141-4183-a6f4-11297e0384c0","Type":"ContainerDied","Data":"ee572aee965b14926b5b23bf24a852b573f5e3390442bdd0ff8460122bdd6d8b"}
Oct 29 22:08:45.994481 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:45.994057    4962 scope.go:122] "RemoveContainer" containerID="1782df9d6fc7b77793e388f65492408da83206592cdea0cc34ebfa08d465e256"
Oct 29 22:08:46.051571 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:46.051543    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"]
Oct 29 22:08:46.063002 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:46.062980    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-380eea-openshift-default-b4d48cf57-c8n25"]
Oct 29 22:08:46.959854 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:46.959819    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"]
Oct 29 22:08:46.960100 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:46.960061    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6" podUID="ed7fcffb-6974-4e95-b689-bc8eb07d9dc7" containerName="maas-api" containerID="cri-o://135a7912771f066cceb519f4c7f0eac1cf5a0fd2359c9f3f7eb1a27239cadaf5" gracePeriod=30
Oct 29 22:08:47.060955 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:47.060923    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1106f45f-cdca-428f-8a2e-dc402d199ae8" path="/var/lib/kubelet/pods/1106f45f-cdca-428f-8a2e-dc402d199ae8/volumes"
Oct 29 22:08:47.061667 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:47.061647    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b3220a5-a632-40bd-b9ad-668bb2b5b237" path="/var/lib/kubelet/pods/9b3220a5-a632-40bd-b9ad-668bb2b5b237/volumes"
Oct 29 22:08:47.062308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:47.062278    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a93430b5-4141-4183-a6f4-11297e0384c0" path="/var/lib/kubelet/pods/a93430b5-4141-4183-a6f4-11297e0384c0/volumes"
Oct 29 22:08:47.302548 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:47.302522    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"
Oct 29 22:08:47.397791 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:47.397762    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-maas-api-tls\") pod \"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7\" (UID: \"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7\") "
Oct 29 22:08:47.398063 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:47.398036    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-openshift-service-ca\") pod \"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7\" (UID: \"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7\") "
Oct 29 22:08:47.398222 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:47.398159    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-kube-api-access-c9j6t\" (UniqueName: \"kubernetes.io/projected/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-kube-api-access-c9j6t\") pod \"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7\" (UID: \"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7\") "
Oct 29 22:08:47.398304 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:47.398270    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-metrics-certs\") pod \"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7\" (UID: \"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7\") "
Oct 29 22:08:47.398960 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:47.398935    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-openshift-service-ca" pod "ed7fcffb-6974-4e95-b689-bc8eb07d9dc7" (UID: "ed7fcffb-6974-4e95-b689-bc8eb07d9dc7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:08:47.399177 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:47.399150    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:47.400220 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:47.400194    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-maas-api-tls" pod "ed7fcffb-6974-4e95-b689-bc8eb07d9dc7" (UID: "ed7fcffb-6974-4e95-b689-bc8eb07d9dc7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:08:47.400918 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:47.400893    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-kube-api-access-c9j6t" pod "ed7fcffb-6974-4e95-b689-bc8eb07d9dc7" (UID: "ed7fcffb-6974-4e95-b689-bc8eb07d9dc7"). InnerVolumeSpecName "kube-api-access-c9j6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:08:47.401008 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:47.400959    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-metrics-certs" pod "ed7fcffb-6974-4e95-b689-bc8eb07d9dc7" (UID: "ed7fcffb-6974-4e95-b689-bc8eb07d9dc7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:08:47.501202 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:47.501136    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9j6t\" (UniqueName: \"kubernetes.io/projected/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-kube-api-access-c9j6t\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:47.501202 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:47.501169    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:47.501202 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:47.501186    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:08:47.736323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:47.716517    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b"
Oct 29 22:08:47.736323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:47.735777    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9"
Oct 29 22:08:48.015679 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.015639    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw"]
Oct 29 22:08:48.017186 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.017155    4962 cpu_manager.go:414] "containerMap: removing container" podUID="a93430b5-4141-4183-a6f4-11297e0384c0" containerName="istio-proxy"
Oct 29 22:08:48.017186 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.017182    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a93430b5-4141-4183-a6f4-11297e0384c0" containerName="istio-proxy"
Oct 29 22:08:48.017413 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.017228    4962 cpu_manager.go:414] "containerMap: removing container" podUID="1106f45f-cdca-428f-8a2e-dc402d199ae8" containerName="istio-proxy"
Oct 29 22:08:48.017413 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.017237    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1106f45f-cdca-428f-8a2e-dc402d199ae8" containerName="istio-proxy"
Oct 29 22:08:48.017413 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.017313    4962 cpu_manager.go:414] "containerMap: removing container" podUID="ed7fcffb-6974-4e95-b689-bc8eb07d9dc7" containerName="maas-api"
Oct 29 22:08:48.017413 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.017323    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed7fcffb-6974-4e95-b689-bc8eb07d9dc7" containerName="maas-api"
Oct 29 22:08:48.017413 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.017346    4962 cpu_manager.go:414] "containerMap: removing container" podUID="9b3220a5-a632-40bd-b9ad-668bb2b5b237" containerName="istio-proxy"
Oct 29 22:08:48.017413 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.017354    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b3220a5-a632-40bd-b9ad-668bb2b5b237" containerName="istio-proxy"
Oct 29 22:08:48.017740 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.017531    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="a93430b5-4141-4183-a6f4-11297e0384c0" containerName="istio-proxy"
Oct 29 22:08:48.017740 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.017552    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="1106f45f-cdca-428f-8a2e-dc402d199ae8" containerName="istio-proxy"
Oct 29 22:08:48.017740 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.017571    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed7fcffb-6974-4e95-b689-bc8eb07d9dc7" containerName="maas-api"
Oct 29 22:08:48.017740 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.017583    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b3220a5-a632-40bd-b9ad-668bb2b5b237" containerName="istio-proxy"
Oct 29 22:08:48.022527 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.022501    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw"
Oct 29 22:08:48.040257 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.040225    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw"]
Oct 29 22:08:48.050268 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.041281    4962 generic.go:356] "Generic (PLEG): container finished" podID="ed7fcffb-6974-4e95-b689-bc8eb07d9dc7" containerID="135a7912771f066cceb519f4c7f0eac1cf5a0fd2359c9f3f7eb1a27239cadaf5" exitCode=0
Oct 29 22:08:48.050268 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.041621    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6" event={"ID":"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7","Type":"ContainerDied","Data":"135a7912771f066cceb519f4c7f0eac1cf5a0fd2359c9f3f7eb1a27239cadaf5"}
Oct 29 22:08:48.050268 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.041666    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6" event={"ID":"ed7fcffb-6974-4e95-b689-bc8eb07d9dc7","Type":"ContainerDied","Data":"9cd04fde088f8d7e43690ee4d7d8f1277cf95e1e998b165feee440809bcd417e"}
Oct 29 22:08:48.050268 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.041694    4962 scope.go:122] "RemoveContainer" containerID="135a7912771f066cceb519f4c7f0eac1cf5a0fd2359c9f3f7eb1a27239cadaf5"
Oct 29 22:08:48.050268 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.041855    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"
Oct 29 22:08:48.050268 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.041892    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9ae129-serving-cert\""
Oct 29 22:08:48.071500 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.071477    4962 scope.go:122] "RemoveContainer" containerID="135a7912771f066cceb519f4c7f0eac1cf5a0fd2359c9f3f7eb1a27239cadaf5"
Oct 29 22:08:48.071862 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:08:48.071834    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"135a7912771f066cceb519f4c7f0eac1cf5a0fd2359c9f3f7eb1a27239cadaf5\": container with ID starting with 135a7912771f066cceb519f4c7f0eac1cf5a0fd2359c9f3f7eb1a27239cadaf5 not found: ID does not exist" containerID="135a7912771f066cceb519f4c7f0eac1cf5a0fd2359c9f3f7eb1a27239cadaf5"
Oct 29 22:08:48.071923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.071868    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"135a7912771f066cceb519f4c7f0eac1cf5a0fd2359c9f3f7eb1a27239cadaf5"} err="failed to get container status \"135a7912771f066cceb519f4c7f0eac1cf5a0fd2359c9f3f7eb1a27239cadaf5\": rpc error: code = NotFound desc = could not find container \"135a7912771f066cceb519f4c7f0eac1cf5a0fd2359c9f3f7eb1a27239cadaf5\": container with ID starting with 135a7912771f066cceb519f4c7f0eac1cf5a0fd2359c9f3f7eb1a27239cadaf5 not found: ID does not exist"
Oct 29 22:08:48.123002 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.122972    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ffbb691-305b-4414-9e01-81aa65c30fa5-metrics-certs\") pod \"maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw\" (UID: \"8ffbb691-305b-4414-9e01-81aa65c30fa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw"
Oct 29 22:08:48.123220 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.123199    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w758l\" (UniqueName: \"kubernetes.io/projected/8ffbb691-305b-4414-9e01-81aa65c30fa5-kube-api-access-w758l\") pod \"maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw\" (UID: \"8ffbb691-305b-4414-9e01-81aa65c30fa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw"
Oct 29 22:08:48.123376 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.123357    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ffbb691-305b-4414-9e01-81aa65c30fa5-openshift-service-ca\") pod \"maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw\" (UID: \"8ffbb691-305b-4414-9e01-81aa65c30fa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw"
Oct 29 22:08:48.123471 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.123453    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ffbb691-305b-4414-9e01-81aa65c30fa5-maas-api-tls\") pod \"maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw\" (UID: \"8ffbb691-305b-4414-9e01-81aa65c30fa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw"
Oct 29 22:08:48.218311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.202405    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"]
Oct 29 22:08:48.262320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.258461    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w758l\" (UniqueName: \"kubernetes.io/projected/8ffbb691-305b-4414-9e01-81aa65c30fa5-kube-api-access-w758l\") pod \"maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw\" (UID: \"8ffbb691-305b-4414-9e01-81aa65c30fa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw"
Oct 29 22:08:48.262320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.258624    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ffbb691-305b-4414-9e01-81aa65c30fa5-openshift-service-ca\") pod \"maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw\" (UID: \"8ffbb691-305b-4414-9e01-81aa65c30fa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw"
Oct 29 22:08:48.262320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.258784    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ffbb691-305b-4414-9e01-81aa65c30fa5-maas-api-tls\") pod \"maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw\" (UID: \"8ffbb691-305b-4414-9e01-81aa65c30fa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw"
Oct 29 22:08:48.262320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.258956    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ffbb691-305b-4414-9e01-81aa65c30fa5-metrics-certs\") pod \"maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw\" (UID: \"8ffbb691-305b-4414-9e01-81aa65c30fa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw"
Oct 29 22:08:48.282238 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.281732    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-734480-85c4b77cdb-m9rh6"]
Oct 29 22:08:48.330315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.329301    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ffbb691-305b-4414-9e01-81aa65c30fa5-maas-api-tls\") pod \"maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw\" (UID: \"8ffbb691-305b-4414-9e01-81aa65c30fa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw"
Oct 29 22:08:48.338382 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.331169    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ffbb691-305b-4414-9e01-81aa65c30fa5-openshift-service-ca\") pod \"maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw\" (UID: \"8ffbb691-305b-4414-9e01-81aa65c30fa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw"
Oct 29 22:08:48.349053 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.349020    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ffbb691-305b-4414-9e01-81aa65c30fa5-metrics-certs\") pod \"maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw\" (UID: \"8ffbb691-305b-4414-9e01-81aa65c30fa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw"
Oct 29 22:08:48.357895 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.352008    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w758l\" (UniqueName: \"kubernetes.io/projected/8ffbb691-305b-4414-9e01-81aa65c30fa5-kube-api-access-w758l\") pod \"maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw\" (UID: \"8ffbb691-305b-4414-9e01-81aa65c30fa5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw"
Oct 29 22:08:48.358514 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.358434    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw"
Oct 29 22:08:48.769002 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:48.768976    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw"]
Oct 29 22:08:48.769344 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:08:48.769280    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ffbb691_305b_4414_9e01_81aa65c30fa5.slice/crio-18798c951840536d1e90ae93daa940518f3b455c1ffcbb982398172cff79cec0 WatchSource:0}: Error finding container 18798c951840536d1e90ae93daa940518f3b455c1ffcbb982398172cff79cec0: Status 404 returned error can't find the container with id 18798c951840536d1e90ae93daa940518f3b455c1ffcbb982398172cff79cec0
Oct 29 22:08:49.126533 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:49.126452    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed7fcffb-6974-4e95-b689-bc8eb07d9dc7" path="/var/lib/kubelet/pods/ed7fcffb-6974-4e95-b689-bc8eb07d9dc7/volumes"
Oct 29 22:08:49.127044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:49.127018    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw" event={"ID":"8ffbb691-305b-4414-9e01-81aa65c30fa5","Type":"ContainerStarted","Data":"18798c951840536d1e90ae93daa940518f3b455c1ffcbb982398172cff79cec0"}
Oct 29 22:08:50.131987 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:50.131950    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw" event={"ID":"8ffbb691-305b-4414-9e01-81aa65c30fa5","Type":"ContainerStarted","Data":"0025361fd9fb5c661e6468b977161a89901cd3b8ab1c99aaada63585c53b3428"}
Oct 29 22:08:50.132431 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:50.132013    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw"
Oct 29 22:08:50.190425 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:50.190379    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw" podStartSLOduration=2.7401546789999998 podStartE2EDuration="3.190365288s" totalImagesPullingTime="450.210609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:08:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:08:50.188501907 +0000 UTC m=+1351.261458873" watchObservedRunningTime="2026-10-29 22:08:50.190365288 +0000 UTC m=+1351.263322254"
Oct 29 22:08:56.161866 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:08:56.161835    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw"
Oct 29 22:09:03.703381 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:03.703338    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp"]
Oct 29 22:09:03.703872 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:03.703589    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp" podUID="6e00f0b9-3982-424a-8c78-7037bcda1f79" containerName="maas-api" containerID="cri-o://d2d6347a492ec3a65b34d4b3cc153cf0a095101eb5ddf24d852f5db521bd90ae" gracePeriod=30
Oct 29 22:09:04.425632 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.425599    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp"
Oct 29 22:09:04.528311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.525014    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg"]
Oct 29 22:09:04.528311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.526936    4962 cpu_manager.go:414] "containerMap: removing container" podUID="6e00f0b9-3982-424a-8c78-7037bcda1f79" containerName="maas-api"
Oct 29 22:09:04.528311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.526960    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e00f0b9-3982-424a-8c78-7037bcda1f79" containerName="maas-api"
Oct 29 22:09:04.528311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.527264    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e00f0b9-3982-424a-8c78-7037bcda1f79" containerName="maas-api"
Oct 29 22:09:04.540311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.538900    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e00f0b9-3982-424a-8c78-7037bcda1f79-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e00f0b9-3982-424a-8c78-7037bcda1f79-maas-api-tls\") pod \"6e00f0b9-3982-424a-8c78-7037bcda1f79\" (UID: \"6e00f0b9-3982-424a-8c78-7037bcda1f79\") "
Oct 29 22:09:04.540311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.538984    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e00f0b9-3982-424a-8c78-7037bcda1f79-kube-api-access-hddt6\" (UniqueName: \"kubernetes.io/projected/6e00f0b9-3982-424a-8c78-7037bcda1f79-kube-api-access-hddt6\") pod \"6e00f0b9-3982-424a-8c78-7037bcda1f79\" (UID: \"6e00f0b9-3982-424a-8c78-7037bcda1f79\") "
Oct 29 22:09:04.540311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.539038    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e00f0b9-3982-424a-8c78-7037bcda1f79-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e00f0b9-3982-424a-8c78-7037bcda1f79-metrics-certs\") pod \"6e00f0b9-3982-424a-8c78-7037bcda1f79\" (UID: \"6e00f0b9-3982-424a-8c78-7037bcda1f79\") "
Oct 29 22:09:04.540311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.539437    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e00f0b9-3982-424a-8c78-7037bcda1f79-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e00f0b9-3982-424a-8c78-7037bcda1f79-openshift-service-ca\") pod \"6e00f0b9-3982-424a-8c78-7037bcda1f79\" (UID: \"6e00f0b9-3982-424a-8c78-7037bcda1f79\") "
Oct 29 22:09:04.542492 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.542459    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e00f0b9-3982-424a-8c78-7037bcda1f79-openshift-service-ca" pod "6e00f0b9-3982-424a-8c78-7037bcda1f79" (UID: "6e00f0b9-3982-424a-8c78-7037bcda1f79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:09:04.543641 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.543619    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg"
Oct 29 22:09:04.563379 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.562560    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a9e67f-serving-cert\""
Oct 29 22:09:04.576129 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.573519    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg"]
Oct 29 22:09:04.576129 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.573841    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e00f0b9-3982-424a-8c78-7037bcda1f79-maas-api-tls" pod "6e00f0b9-3982-424a-8c78-7037bcda1f79" (UID: "6e00f0b9-3982-424a-8c78-7037bcda1f79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:09:04.576129 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.575780    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e00f0b9-3982-424a-8c78-7037bcda1f79-kube-api-access-hddt6" pod "6e00f0b9-3982-424a-8c78-7037bcda1f79" (UID: "6e00f0b9-3982-424a-8c78-7037bcda1f79"). InnerVolumeSpecName "kube-api-access-hddt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:09:04.585358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.583774    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e00f0b9-3982-424a-8c78-7037bcda1f79-metrics-certs" pod "6e00f0b9-3982-424a-8c78-7037bcda1f79" (UID: "6e00f0b9-3982-424a-8c78-7037bcda1f79"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:09:04.663155 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.655122    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dec9c57a-18ef-4c9c-8dd1-6876994b6578-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg\" (UID: \"dec9c57a-18ef-4c9c-8dd1-6876994b6578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg"
Oct 29 22:09:04.663155 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.655333    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss44p\" (UniqueName: \"kubernetes.io/projected/dec9c57a-18ef-4c9c-8dd1-6876994b6578-kube-api-access-ss44p\") pod \"maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg\" (UID: \"dec9c57a-18ef-4c9c-8dd1-6876994b6578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg"
Oct 29 22:09:04.663155 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.655386    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dec9c57a-18ef-4c9c-8dd1-6876994b6578-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg\" (UID: \"dec9c57a-18ef-4c9c-8dd1-6876994b6578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg"
Oct 29 22:09:04.663155 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.655481    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec9c57a-18ef-4c9c-8dd1-6876994b6578-metrics-certs\") pod \"maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg\" (UID: \"dec9c57a-18ef-4c9c-8dd1-6876994b6578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg"
Oct 29 22:09:04.663155 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.655985    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e00f0b9-3982-424a-8c78-7037bcda1f79-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:09:04.663155 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.656008    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hddt6\" (UniqueName: \"kubernetes.io/projected/6e00f0b9-3982-424a-8c78-7037bcda1f79-kube-api-access-hddt6\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:09:04.663155 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.656025    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e00f0b9-3982-424a-8c78-7037bcda1f79-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:09:04.663155 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.656046    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e00f0b9-3982-424a-8c78-7037bcda1f79-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:09:04.719671 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.718533    4962 generic.go:356] "Generic (PLEG): container finished" podID="6e00f0b9-3982-424a-8c78-7037bcda1f79" containerID="d2d6347a492ec3a65b34d4b3cc153cf0a095101eb5ddf24d852f5db521bd90ae" exitCode=0
Oct 29 22:09:04.719671 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.718788    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp" event={"ID":"6e00f0b9-3982-424a-8c78-7037bcda1f79","Type":"ContainerDied","Data":"d2d6347a492ec3a65b34d4b3cc153cf0a095101eb5ddf24d852f5db521bd90ae"}
Oct 29 22:09:04.719671 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.718837    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp" event={"ID":"6e00f0b9-3982-424a-8c78-7037bcda1f79","Type":"ContainerDied","Data":"e9cd9e4b59c703af805664737fbda7059ecf834d250843802b9c485f5d3bcd15"}
Oct 29 22:09:04.719671 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.718865    4962 scope.go:122] "RemoveContainer" containerID="d2d6347a492ec3a65b34d4b3cc153cf0a095101eb5ddf24d852f5db521bd90ae"
Oct 29 22:09:04.719671 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.719019    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp"
Oct 29 22:09:04.748605 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.748298    4962 scope.go:122] "RemoveContainer" containerID="d2d6347a492ec3a65b34d4b3cc153cf0a095101eb5ddf24d852f5db521bd90ae"
Oct 29 22:09:04.748694 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:09:04.748643    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2d6347a492ec3a65b34d4b3cc153cf0a095101eb5ddf24d852f5db521bd90ae\": container with ID starting with d2d6347a492ec3a65b34d4b3cc153cf0a095101eb5ddf24d852f5db521bd90ae not found: ID does not exist" containerID="d2d6347a492ec3a65b34d4b3cc153cf0a095101eb5ddf24d852f5db521bd90ae"
Oct 29 22:09:04.748760 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.748684    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2d6347a492ec3a65b34d4b3cc153cf0a095101eb5ddf24d852f5db521bd90ae"} err="failed to get container status \"d2d6347a492ec3a65b34d4b3cc153cf0a095101eb5ddf24d852f5db521bd90ae\": rpc error: code = NotFound desc = could not find container \"d2d6347a492ec3a65b34d4b3cc153cf0a095101eb5ddf24d852f5db521bd90ae\": container with ID starting with d2d6347a492ec3a65b34d4b3cc153cf0a095101eb5ddf24d852f5db521bd90ae not found: ID does not exist"
Oct 29 22:09:04.757609 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.757582    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dec9c57a-18ef-4c9c-8dd1-6876994b6578-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg\" (UID: \"dec9c57a-18ef-4c9c-8dd1-6876994b6578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg"
Oct 29 22:09:04.757757 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.757738    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss44p\" (UniqueName: \"kubernetes.io/projected/dec9c57a-18ef-4c9c-8dd1-6876994b6578-kube-api-access-ss44p\") pod \"maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg\" (UID: \"dec9c57a-18ef-4c9c-8dd1-6876994b6578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg"
Oct 29 22:09:04.757838 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.757786    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dec9c57a-18ef-4c9c-8dd1-6876994b6578-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg\" (UID: \"dec9c57a-18ef-4c9c-8dd1-6876994b6578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg"
Oct 29 22:09:04.757901 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.757880    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec9c57a-18ef-4c9c-8dd1-6876994b6578-metrics-certs\") pod \"maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg\" (UID: \"dec9c57a-18ef-4c9c-8dd1-6876994b6578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg"
Oct 29 22:09:04.758965 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.758943    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dec9c57a-18ef-4c9c-8dd1-6876994b6578-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg\" (UID: \"dec9c57a-18ef-4c9c-8dd1-6876994b6578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg"
Oct 29 22:09:04.760980 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.760957    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec9c57a-18ef-4c9c-8dd1-6876994b6578-metrics-certs\") pod \"maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg\" (UID: \"dec9c57a-18ef-4c9c-8dd1-6876994b6578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg"
Oct 29 22:09:04.761268 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.761246    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dec9c57a-18ef-4c9c-8dd1-6876994b6578-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg\" (UID: \"dec9c57a-18ef-4c9c-8dd1-6876994b6578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg"
Oct 29 22:09:04.784122 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.784097    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp"]
Oct 29 22:09:04.787455 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.786044    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss44p\" (UniqueName: \"kubernetes.io/projected/dec9c57a-18ef-4c9c-8dd1-6876994b6578-kube-api-access-ss44p\") pod \"maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg\" (UID: \"dec9c57a-18ef-4c9c-8dd1-6876994b6578\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg"
Oct 29 22:09:04.881392 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.880033    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg"
Oct 29 22:09:04.940311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:04.935224    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a26f29-7655d8cb5b-m2rpp"]
Oct 29 22:09:05.064119 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:05.063138    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e00f0b9-3982-424a-8c78-7037bcda1f79" path="/var/lib/kubelet/pods/6e00f0b9-3982-424a-8c78-7037bcda1f79/volumes"
Oct 29 22:09:05.317859 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:09:05.317828    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec9c57a_18ef_4c9c_8dd1_6876994b6578.slice/crio-b778034116c39b391c1a527b7571fca58d90f898db6514b5a0030be060839ac7 WatchSource:0}: Error finding container b778034116c39b391c1a527b7571fca58d90f898db6514b5a0030be060839ac7: Status 404 returned error can't find the container with id b778034116c39b391c1a527b7571fca58d90f898db6514b5a0030be060839ac7
Oct 29 22:09:05.318117 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:05.318095    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg"]
Oct 29 22:09:05.897346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:05.897302    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg" event={"ID":"dec9c57a-18ef-4c9c-8dd1-6876994b6578","Type":"ContainerStarted","Data":"b778034116c39b391c1a527b7571fca58d90f898db6514b5a0030be060839ac7"}
Oct 29 22:09:06.910831 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:06.910788    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg" event={"ID":"dec9c57a-18ef-4c9c-8dd1-6876994b6578","Type":"ContainerStarted","Data":"7c44bb504d9f825be4e51db76182f550c3f9c33201d91d03bb5f28ea0a730fb8"}
Oct 29 22:09:06.911252 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:06.910853    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg"
Oct 29 22:09:06.998300 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:06.998251    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg" podStartSLOduration=2.42493825 podStartE2EDuration="2.998238235s" totalImagesPullingTime="573.299985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:06.994344581 +0000 UTC m=+1368.067301544" watchObservedRunningTime="2026-10-29 22:09:06.998238235 +0000 UTC m=+1368.071195199"
Oct 29 22:09:09.726723 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:09:09.726687    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:09:12.925834 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:12.925807    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg"
Oct 29 22:09:14.438310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.432521    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"]
Oct 29 22:09:14.450862 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.450824    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.557334 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.541604    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f82846-openshift-default-dockercfg-9s8bb\""
Oct 29 22:09:14.706429 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.687985    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-data\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.706429 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.688106    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-workload-certs\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.706429 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.688186    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-envoy\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.706429 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.688219    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4q9f\" (UniqueName: \"kubernetes.io/projected/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-kube-api-access-q4q9f\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.706429 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.688347    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-workload-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.706429 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.688410    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-token\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.706429 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.688500    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-credential-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.706429 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.688569    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istiod-ca-cert\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.706429 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.688601    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-podinfo\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.744172 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.744138    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"]
Oct 29 22:09:14.798362 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.798039    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-data\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.798362 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.798160    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-workload-certs\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.798362 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.798224    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-envoy\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.798362 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.798252    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4q9f\" (UniqueName: \"kubernetes.io/projected/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-kube-api-access-q4q9f\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.798362 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.798367    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-workload-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.798750 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.798421    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-token\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.798750 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.798492    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-credential-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.798750 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.798546    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istiod-ca-cert\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.798750 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.798574    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-podinfo\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.823320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.822664    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-workload-certs\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.830432 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.824841    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-credential-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.830432 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.825583    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istiod-ca-cert\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.830432 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.829454    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-workload-socket\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.841934 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.835217    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-podinfo\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.869341 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.858079    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-data\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.881165 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.880570    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4q9f\" (UniqueName: \"kubernetes.io/projected/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-kube-api-access-q4q9f\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.884231 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.884112    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-token\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:14.986250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:14.979672    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-envoy\") pod \"e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") " pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:15.087713 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:15.077860    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:15.494140 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:09:15.494109    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaea6cb2_afaf_49a8_81ab_bb9c3f699e35.slice/crio-5692a96c5ff999f2e5aba13b787e198a6e342e104a1f220d35527c7804097022 WatchSource:0}: Error finding container 5692a96c5ff999f2e5aba13b787e198a6e342e104a1f220d35527c7804097022: Status 404 returned error can't find the container with id 5692a96c5ff999f2e5aba13b787e198a6e342e104a1f220d35527c7804097022
Oct 29 22:09:15.494546 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:15.494523    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"]
Oct 29 22:09:15.496779 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:15.496751    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:09:15.496828 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:15.496816    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:09:15.496865 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:15.496848    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:09:16.175722 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:16.175679    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm" event={"ID":"faea6cb2-afaf-49a8-81ab-bb9c3f699e35","Type":"ContainerStarted","Data":"44c462ca217edfa89d757be4543e0b8ba0dc40798afa07384667979bc9d4b870"}
Oct 29 22:09:16.175893 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:16.175745    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm" event={"ID":"faea6cb2-afaf-49a8-81ab-bb9c3f699e35","Type":"ContainerStarted","Data":"5692a96c5ff999f2e5aba13b787e198a6e342e104a1f220d35527c7804097022"}
Oct 29 22:09:16.237194 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:16.237136    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm" podStartSLOduration=2.237120637 podStartE2EDuration="2.237120637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:16.233165691 +0000 UTC m=+1377.306122660" watchObservedRunningTime="2026-10-29 22:09:16.237120637 +0000 UTC m=+1377.310077600"
Oct 29 22:09:16.555417 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:16.551347    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk"]
Oct 29 22:09:16.555417 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:16.553188    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk" podUID="91215c4c-55d2-4472-b58b-5612e1012574" containerName="maas-api" containerID="cri-o://6d78cb9ac9283b9c85c9b62a0c123a5ed11c4af8749237588172206b4a7b4800" gracePeriod=30
Oct 29 22:09:16.893452 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:16.893425    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk"
Oct 29 22:09:17.020389 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.020357    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91215c4c-55d2-4472-b58b-5612e1012574-kube-api-access-p6f6z\" (UniqueName: \"kubernetes.io/projected/91215c4c-55d2-4472-b58b-5612e1012574-kube-api-access-p6f6z\") pod \"91215c4c-55d2-4472-b58b-5612e1012574\" (UID: \"91215c4c-55d2-4472-b58b-5612e1012574\") "
Oct 29 22:09:17.020550 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.020426    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91215c4c-55d2-4472-b58b-5612e1012574-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91215c4c-55d2-4472-b58b-5612e1012574-maas-api-tls\") pod \"91215c4c-55d2-4472-b58b-5612e1012574\" (UID: \"91215c4c-55d2-4472-b58b-5612e1012574\") "
Oct 29 22:09:17.020550 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.020467    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91215c4c-55d2-4472-b58b-5612e1012574-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91215c4c-55d2-4472-b58b-5612e1012574-openshift-service-ca\") pod \"91215c4c-55d2-4472-b58b-5612e1012574\" (UID: \"91215c4c-55d2-4472-b58b-5612e1012574\") "
Oct 29 22:09:17.020688 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.020607    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91215c4c-55d2-4472-b58b-5612e1012574-metrics-certs\" (UniqueName: \"kubernetes.io/secret/91215c4c-55d2-4472-b58b-5612e1012574-metrics-certs\") pod \"91215c4c-55d2-4472-b58b-5612e1012574\" (UID: \"91215c4c-55d2-4472-b58b-5612e1012574\") "
Oct 29 22:09:17.020833 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.020809    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91215c4c-55d2-4472-b58b-5612e1012574-openshift-service-ca" pod "91215c4c-55d2-4472-b58b-5612e1012574" (UID: "91215c4c-55d2-4472-b58b-5612e1012574"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:09:17.021435 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.021411    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91215c4c-55d2-4472-b58b-5612e1012574-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:09:17.022979 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.022913    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91215c4c-55d2-4472-b58b-5612e1012574-kube-api-access-p6f6z" pod "91215c4c-55d2-4472-b58b-5612e1012574" (UID: "91215c4c-55d2-4472-b58b-5612e1012574"). InnerVolumeSpecName "kube-api-access-p6f6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:09:17.023173 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.023131    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91215c4c-55d2-4472-b58b-5612e1012574-maas-api-tls" pod "91215c4c-55d2-4472-b58b-5612e1012574" (UID: "91215c4c-55d2-4472-b58b-5612e1012574"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:09:17.023264 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.023220    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91215c4c-55d2-4472-b58b-5612e1012574-metrics-certs" pod "91215c4c-55d2-4472-b58b-5612e1012574" (UID: "91215c4c-55d2-4472-b58b-5612e1012574"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:09:17.169568 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.085384    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:17.171677 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.170079    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6f6z\" (UniqueName: \"kubernetes.io/projected/91215c4c-55d2-4472-b58b-5612e1012574-kube-api-access-p6f6z\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:09:17.171677 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.170109    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91215c4c-55d2-4472-b58b-5612e1012574-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:09:17.171677 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.170122    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91215c4c-55d2-4472-b58b-5612e1012574-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:09:17.296309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.293022    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:17.440692 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.440602    4962 generic.go:356] "Generic (PLEG): container finished" podID="91215c4c-55d2-4472-b58b-5612e1012574" containerID="6d78cb9ac9283b9c85c9b62a0c123a5ed11c4af8749237588172206b4a7b4800" exitCode=0
Oct 29 22:09:17.446595 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.446551    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk"
Oct 29 22:09:17.454313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.452337    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk" event={"ID":"91215c4c-55d2-4472-b58b-5612e1012574","Type":"ContainerDied","Data":"6d78cb9ac9283b9c85c9b62a0c123a5ed11c4af8749237588172206b4a7b4800"}
Oct 29 22:09:17.454313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.452419    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk" event={"ID":"91215c4c-55d2-4472-b58b-5612e1012574","Type":"ContainerDied","Data":"9ccc7cc8b42175ad83b799720155094a508b90a887bed50bb273dcb26f617c3c"}
Oct 29 22:09:17.454313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.452449    4962 scope.go:122] "RemoveContainer" containerID="6d78cb9ac9283b9c85c9b62a0c123a5ed11c4af8749237588172206b4a7b4800"
Oct 29 22:09:17.454313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.453114    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:17.469861 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.455480    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:09:17.558241 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.556629    4962 scope.go:122] "RemoveContainer" containerID="6d78cb9ac9283b9c85c9b62a0c123a5ed11c4af8749237588172206b4a7b4800"
Oct 29 22:09:17.559434 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:09:17.559351    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d78cb9ac9283b9c85c9b62a0c123a5ed11c4af8749237588172206b4a7b4800\": container with ID starting with 6d78cb9ac9283b9c85c9b62a0c123a5ed11c4af8749237588172206b4a7b4800 not found: ID does not exist" containerID="6d78cb9ac9283b9c85c9b62a0c123a5ed11c4af8749237588172206b4a7b4800"
Oct 29 22:09:17.559637 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.559607    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d78cb9ac9283b9c85c9b62a0c123a5ed11c4af8749237588172206b4a7b4800"} err="failed to get container status \"6d78cb9ac9283b9c85c9b62a0c123a5ed11c4af8749237588172206b4a7b4800\": rpc error: code = NotFound desc = could not find container \"6d78cb9ac9283b9c85c9b62a0c123a5ed11c4af8749237588172206b4a7b4800\": container with ID starting with 6d78cb9ac9283b9c85c9b62a0c123a5ed11c4af8749237588172206b4a7b4800 not found: ID does not exist"
Oct 29 22:09:17.603372 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.603343    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk"]
Oct 29 22:09:17.613676 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:17.613650    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2d53cd-5c79dd6954-2ljqk"]
Oct 29 22:09:19.061427 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:19.061396    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91215c4c-55d2-4472-b58b-5612e1012574" path="/var/lib/kubelet/pods/91215c4c-55d2-4472-b58b-5612e1012574/volumes"
Oct 29 22:09:22.336312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:22.335086    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:22.570315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:22.570164    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:22.679263 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:22.675390    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:22.679263 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:22.675835    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:09:22.695033 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:22.694981    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:22.695630 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:22.695595    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:09:22.823150 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:22.821391    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:23.084461 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:23.084370    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:23.335535 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:23.335424    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:23.337372 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:23.335967    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 22:09:23.538965 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:23.537988    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:09:23.540745 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:23.540703    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:23.741315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:23.737755    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:23.766592 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:23.751238    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:23.778010 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:23.777365    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 22:09:23.778010 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:23.777478    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:09:23.832734 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:23.756594    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:09:23.832734 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:23.756622    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:09:23.832734 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:23.804873    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:09:23.832734 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:23.804936    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 22:09:23.832734 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:23.752096    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:09:23.843412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:23.839379    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:09:23.843412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:23.839703    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:09:24.136331 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.136221    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:09:24.156662 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.156630    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 22:09:24.156897 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.156873    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg"]
Oct 29 22:09:24.179373 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.158361    4962 cpu_manager.go:414] "containerMap: removing container" podUID="91215c4c-55d2-4472-b58b-5612e1012574" containerName="maas-api"
Oct 29 22:09:24.179594 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.179575    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91215c4c-55d2-4472-b58b-5612e1012574" containerName="maas-api"
Oct 29 22:09:24.180000 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.179979    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="91215c4c-55d2-4472-b58b-5612e1012574" containerName="maas-api"
Oct 29 22:09:24.247311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.224802    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg"
Oct 29 22:09:24.288941 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.237517    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 22:09:24.369856 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.302705    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f82846-dockercfg-rtf4m\""
Oct 29 22:09:24.369856 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.325103    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f82846\""
Oct 29 22:09:24.369856 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.325550    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:24.369856 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.344737    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:24.369856 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.363422    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:24.595333 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.560428    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:09:24.595333 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.560909    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:09:24.648311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.630834    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:24.648311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.631274    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:09:24.778893 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.778801    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:09:24.789309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.780029    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:24.834548 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.834506    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:24.835728 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.835707    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 22:09:24.836613 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.836586    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:24.886337 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.884863    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:24.899680 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.899648    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f592a322-7cc5-41d0-8212-18797b23b843-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg\" (UID: \"f592a322-7cc5-41d0-8212-18797b23b843\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg"
Oct 29 22:09:24.900141 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.900113    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwtz6\" (UniqueName: \"kubernetes.io/projected/f592a322-7cc5-41d0-8212-18797b23b843-kube-api-access-gwtz6\") pod \"payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg\" (UID: \"f592a322-7cc5-41d0-8212-18797b23b843\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg"
Oct 29 22:09:24.956311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:24.930833    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg"]
Oct 29 22:09:25.200313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:25.181762    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f592a322-7cc5-41d0-8212-18797b23b843-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg\" (UID: \"f592a322-7cc5-41d0-8212-18797b23b843\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg"
Oct 29 22:09:25.201073 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:25.201032    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwtz6\" (UniqueName: \"kubernetes.io/projected/f592a322-7cc5-41d0-8212-18797b23b843-kube-api-access-gwtz6\") pod \"payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg\" (UID: \"f592a322-7cc5-41d0-8212-18797b23b843\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg"
Oct 29 22:09:25.228312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:25.203024    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f592a322-7cc5-41d0-8212-18797b23b843-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg\" (UID: \"f592a322-7cc5-41d0-8212-18797b23b843\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg"
Oct 29 22:09:25.446353 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:25.443027    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:25.457082 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:25.442942    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:25.489416 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:25.443095    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:09:25.584314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:25.580828    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:25.684309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:25.677119    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:25.753461 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:25.744364    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:25.754870 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:25.754823    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwtz6\" (UniqueName: \"kubernetes.io/projected/f592a322-7cc5-41d0-8212-18797b23b843-kube-api-access-gwtz6\") pod \"payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg\" (UID: \"f592a322-7cc5-41d0-8212-18797b23b843\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg"
Oct 29 22:09:25.884404 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:25.779518    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:25.915359 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:25.805810    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:25.917238 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:25.848382    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:25.930680 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:25.848423    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:25.978308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:25.885453    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:26.537559 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:26.537512    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:26.594339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:26.578813    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg"
Oct 29 22:09:26.594339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:26.591215    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:26.758057 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:26.758007    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:09:27.064352 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:27.056299    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch"]
Oct 29 22:09:27.110320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:26.811512    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:09:27.110320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:27.091836    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:09:27.110320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:27.092485    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:27.186456 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:27.186409    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:09:27.204979 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:26.866932    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:09:27.274313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:27.260325    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:27.274313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:27.260389    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds" podUID="2a71a069-94f9-4736-8b9f-d43292554254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:27.279106 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:27.279068    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:27.285837 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:27.282991    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks" podUID="a85f7d35-4bff-45d0-a392-2ecf056c2051" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:27.353648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:27.349524    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg" podUID="70e1bafe-4b93-44d7-a179-44d5a82fd069" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:27.526572 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:27.526506    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:27.599308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:27.598797    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:09:27.901429 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:27.901329    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:27.902618 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:27.902592    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch"
Oct 29 22:09:27.905385 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:27.905343    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg" podUID="16540bea-446b-46d9-9cde-4c9c34785ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:27.940797 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:27.940745    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:28.075311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:28.073921    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:28.200389 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:28.085866    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zssk7\" (UniqueName: \"kubernetes.io/projected/5c4bfc57-2b32-410a-ab06-610e5749ab0e-kube-api-access-zssk7\") pod \"payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch\" (UID: \"5c4bfc57-2b32-410a-ab06-610e5749ab0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch"
Oct 29 22:09:28.248897 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:28.106475    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:28.253371 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:28.108248    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:09:28.271748 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:28.108333    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:28.288519 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:28.108354    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:09:28.315646 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:28.191711    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:28.316231 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:28.231461    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:09:28.341705 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:28.270496    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c4bfc57-2b32-410a-ab06-610e5749ab0e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch\" (UID: \"5c4bfc57-2b32-410a-ab06-610e5749ab0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch"
Oct 29 22:09:28.342525 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:28.288766    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:28.421314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:28.417869    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 22:09:28.421583 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:28.421559    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch"]
Oct 29 22:09:28.458548 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:28.458449    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c4bfc57-2b32-410a-ab06-610e5749ab0e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch\" (UID: \"5c4bfc57-2b32-410a-ab06-610e5749ab0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch"
Oct 29 22:09:28.459391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:28.459363    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zssk7\" (UniqueName: \"kubernetes.io/projected/5c4bfc57-2b32-410a-ab06-610e5749ab0e-kube-api-access-zssk7\") pod \"payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch\" (UID: \"5c4bfc57-2b32-410a-ab06-610e5749ab0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch"
Oct 29 22:09:28.460708 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:28.460686    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c4bfc57-2b32-410a-ab06-610e5749ab0e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch\" (UID: \"5c4bfc57-2b32-410a-ab06-610e5749ab0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch"
Oct 29 22:09:28.607783 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:28.606730    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zssk7\" (UniqueName: \"kubernetes.io/projected/5c4bfc57-2b32-410a-ab06-610e5749ab0e-kube-api-access-zssk7\") pod \"payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch\" (UID: \"5c4bfc57-2b32-410a-ab06-610e5749ab0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch"
Oct 29 22:09:28.607783 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:28.606839    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:28.868400 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:28.864063    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch"
Oct 29 22:09:28.990956 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:28.958988    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:29.037473 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:28.995505    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:09:29.320093 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:29.320050    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:29.518660 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:29.350633    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:29.524699 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:29.361407    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:29.731313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:29.361445    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:29.731313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:29.362598    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:29.766102 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:29.573334    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:29.766102 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:29.655136    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:29.766102 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:29.667056    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:09:30.023436 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:30.023386    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:30.071318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:30.027414    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:30.168579 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:30.168524    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:30.172378 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:30.168827    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57" podUID="67b4a685-bfa5-4a36-adf7-68c7591137c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:30.471888 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:09:30.186669    4962 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.119s"
Oct 29 22:09:30.480394 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:30.409471    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:30.480686 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:30.480641    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp" podUID="5d5e71cf-5078-4917-94b8-2b531b143393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:30.516838 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:30.516484    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded"
Oct 29 22:09:30.518356 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:30.518327    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:30.520234 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:30.520201    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:30.680906 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:30.680858    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:30.738749 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:30.738698    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:30.741815 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:30.741775    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:09:30.742083 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:30.742056    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:09:30.742312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:30.742273    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:09:30.742516 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:30.742494    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:09:30.742748 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:30.742726    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:09:30.830207 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:30.830166    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": context deadline exceeded"
Oct 29 22:09:30.879788 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:30.879414    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:30.879788 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:30.879481    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc" podUID="b72eb074-2855-4dd2-bf0b-e4a42e92d8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:30.931509 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:30.929925    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5"]
Oct 29 22:09:31.037471 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:31.035407    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:31.129314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:31.123481    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:31.209317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:31.209215    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5"
Oct 29 22:09:31.285833 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:31.285801    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b80d5b\""
Oct 29 22:09:31.318114 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:31.302546    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b80d5b-dockercfg-mtcrq\""
Oct 29 22:09:31.528582 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:31.528483    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:09:31.529758 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:31.529720    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:31.596640 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:31.582228    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t2lc\" (UniqueName: \"kubernetes.io/projected/f06268ed-ff58-4564-abd0-54609575a363-kube-api-access-4t2lc\") pod \"payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5\" (UID: \"f06268ed-ff58-4564-abd0-54609575a363\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5"
Oct 29 22:09:31.596640 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:31.582829    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06268ed-ff58-4564-abd0-54609575a363-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5\" (UID: \"f06268ed-ff58-4564-abd0-54609575a363\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5"
Oct 29 22:09:31.686315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:31.681861    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5"]
Oct 29 22:09:31.914745 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:31.845447    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:09:31.957859 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:31.957818    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:32.443161 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:31.988872    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:32.499505 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:31.988935    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:32.499505 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:31.988963    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:32.499505 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:31.989012    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:32.499505 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:32.291604    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded"
Oct 29 22:09:32.499505 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:32.446089    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg" event={"ID":"f592a322-7cc5-41d0-8212-18797b23b843","Type":"ContainerStarted","Data":"21d6050e1fe01b05bc01e9a006ac88420dedb3722ebb72e63f733d7b31da3878"}
Oct 29 22:09:32.581717 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:32.581663    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:32.614382 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:32.591029    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06268ed-ff58-4564-abd0-54609575a363-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5\" (UID: \"f06268ed-ff58-4564-abd0-54609575a363\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5"
Oct 29 22:09:32.614382 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:32.608064    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t2lc\" (UniqueName: \"kubernetes.io/projected/f06268ed-ff58-4564-abd0-54609575a363-kube-api-access-4t2lc\") pod \"payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5\" (UID: \"f06268ed-ff58-4564-abd0-54609575a363\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5"
Oct 29 22:09:32.614382 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:32.592011    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06268ed-ff58-4564-abd0-54609575a363-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5\" (UID: \"f06268ed-ff58-4564-abd0-54609575a363\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5"
Oct 29 22:09:32.648603 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:32.635818    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:09:32.803558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:32.762118    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:32.972319 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:32.936438    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z" podUID="132ed775-c9e6-41f1-b224-fff550c6e073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:32.972319 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:32.895025    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:09:32.972319 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:32.832104    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t2lc\" (UniqueName: \"kubernetes.io/projected/f06268ed-ff58-4564-abd0-54609575a363-kube-api-access-4t2lc\") pod \"payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5\" (UID: \"f06268ed-ff58-4564-abd0-54609575a363\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5"
Oct 29 22:09:32.972319 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:32.907404    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:32.974157 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:32.974115    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:09:33.032895 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:33.032843    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:09:33.121139 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:33.120864    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5"
Oct 29 22:09:33.248312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:33.222549    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:33.248312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:33.222629    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv" podUID="d71de0ec-1eaa-481f-8cee-cb256e330136" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.444501 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:33.444447    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.476091 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:33.476049    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.480722 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:33.480670    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.485417 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:33.485367    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:09:33.507560 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:33.485627    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.646082 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:33.646021    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:09:33.903475 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:33.903375    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg"]
Oct 29 22:09:33.943483 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:33.943435    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.962373 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:33.944272    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:33.966006 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:33.964935    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch" event={"ID":"5c4bfc57-2b32-410a-ab06-610e5749ab0e","Type":"ContainerStarted","Data":"3238db60a092576f5b28733ab84d1d46865f97765f22214a7c92cab1f90ed958"}
Oct 29 22:09:33.981313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:33.969321    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg" event={"ID":"f592a322-7cc5-41d0-8212-18797b23b843","Type":"ContainerStarted","Data":"827840f7af12ff8add91cbb0dd1043bbb2000669b85a5a5d00904d51aa2d23d6"}
Oct 29 22:09:33.983396 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:33.983357    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.040462 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:34.040422    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.119392 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:33.983281    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg"
Oct 29 22:09:34.204747 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:34.204712    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:34.205427 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:34.205361    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc" podUID="35c2b463-e2a6-4b1d-8425-558ca7832668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.220449 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:34.220412    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:34.220715 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:34.220680    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz" podUID="8054bc19-6db7-462f-8af0-335903ce62f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.230560 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:34.230514    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:34.230952 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:34.230913    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt" podUID="730edf47-8b07-477a-acfc-47faaaacfc84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.294468 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:34.274026    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.317558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:34.282391    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:34.317865 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:34.294374    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch"]
Oct 29 22:09:34.318180 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:34.318073    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc" podUID="036f71f6-61b8-469d-a080-d1ab0b45c195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.349863 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:34.349826    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.457155 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:34.457116    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.482326 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:34.466805    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg" podStartSLOduration=10.466781868 podStartE2EDuration="10.466781868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:34.039308824 +0000 UTC m=+1395.112265828" watchObservedRunningTime="2026-10-29 22:09:34.466781868 +0000 UTC m=+1395.539738833"
Oct 29 22:09:34.482326 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:34.477505    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.514537 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:34.510329    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:09:34.537425 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:34.537382    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.543434 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:34.543396    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.606906 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:34.606871    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:34.628646 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:34.628601    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42" podUID="e1897161-0016-400a-9928-8b407f4148e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.651322 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:34.649373    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.946305 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:34.945832    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.973064 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:34.969401    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:34.986171 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:34.985678    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:35.051462 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:35.051208    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch" event={"ID":"5c4bfc57-2b32-410a-ab06-610e5749ab0e","Type":"ContainerStarted","Data":"9f42c8ab44f21d74130238a4f1014bc99f670c8c66a6cd6aab9c41a49f0dc0f4"}
Oct 29 22:09:35.051462 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:35.051332    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch"
Oct 29 22:09:35.052006 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:35.051534    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:35.467523 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:35.464786    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:09:35.494329 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:35.478931    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:09:35.518779 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:35.518737    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:09:35.521116 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:35.521059    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch" podStartSLOduration=11.521039995 podStartE2EDuration="11.521039995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:35.108614613 +0000 UTC m=+1396.181571578" watchObservedRunningTime="2026-10-29 22:09:35.521039995 +0000 UTC m=+1396.593996961"
Oct 29 22:09:35.571144 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:35.557503    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:35.625317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:35.603277    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:35.807890 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:35.797568    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:35.855996 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:35.855941    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:36.115021 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:36.114927    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:09:36.189311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:36.160085    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:09:36.189311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:36.163309    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:36.189311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:36.163692    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:09:36.414497 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:36.388378    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:09:36.414497 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:36.388431    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr" podUID="d91917d2-1f66-4ee4-a312-57398651739e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:36.480417 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:36.480135    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:09:36.560484 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:36.560445    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:36.563382 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:36.563353    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:36.566426 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:36.564373    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:09:36.581156 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:36.580920    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5"]
Oct 29 22:09:36.631381 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:36.621596    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:36.728995 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:36.722562    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 22:09:36.838975 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:36.826273    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:09:37.126307 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:37.118515    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:37.165368 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:37.165223    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:37.168268 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:37.166926    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:09:37.501361 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:37.470856    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:09:37.546465 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:37.545857    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:09:37.600380 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:37.584128    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 22:09:37.603228 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:37.602405    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:09:37.643541 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:37.641724    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 22:09:37.648918 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:37.648694    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:37.732315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:37.728118    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:09:37.732315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:37.729018    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:09:37.732315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:37.730683    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 22:09:37.793405 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:37.789489    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:09:37.793405 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:37.791392    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5" event={"ID":"f06268ed-ff58-4564-abd0-54609575a363","Type":"ContainerStarted","Data":"cfd30bf342c789cbaaab0bcc04f8826bd668b7d035f28867c1bb5909574cea03"}
Oct 29 22:09:37.793405 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:37.791433    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5" event={"ID":"f06268ed-ff58-4564-abd0-54609575a363","Type":"ContainerStarted","Data":"9c19a427d89593d51400bfb7480ae6c4381a8939a6eed2178ed4327c8e57fb79"}
Oct 29 22:09:37.793405 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:37.791461    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5"
Oct 29 22:09:37.835840 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:37.834696    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:09:37.838246 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:37.838220    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 22:09:38.204311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:38.203882    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5" podStartSLOduration=12.203864459 podStartE2EDuration="12.203864459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:38.203751687 +0000 UTC m=+1399.276708653" watchObservedRunningTime="2026-10-29 22:09:38.203864459 +0000 UTC m=+1399.276821424"
Oct 29 22:09:39.725563 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:09:39.725526    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:09:45.704622 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:45.704588    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj"]
Oct 29 22:09:45.709478 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:45.709461    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj"
Oct 29 22:09:45.746042 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:45.746017    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-48c52b\""
Oct 29 22:09:45.746042 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:45.746020    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-48c52b-dockercfg-7njvd\""
Oct 29 22:09:45.751071 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:45.751048    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj"]
Oct 29 22:09:45.753279 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:45.753252    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l96hh\" (UniqueName: \"kubernetes.io/projected/60d52b51-0159-402d-8d66-3c453a0e438f-kube-api-access-l96hh\") pod \"payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj\" (UID: \"60d52b51-0159-402d-8d66-3c453a0e438f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj"
Oct 29 22:09:45.753447 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:45.753426    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60d52b51-0159-402d-8d66-3c453a0e438f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj\" (UID: \"60d52b51-0159-402d-8d66-3c453a0e438f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj"
Oct 29 22:09:45.854660 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:45.854624    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l96hh\" (UniqueName: \"kubernetes.io/projected/60d52b51-0159-402d-8d66-3c453a0e438f-kube-api-access-l96hh\") pod \"payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj\" (UID: \"60d52b51-0159-402d-8d66-3c453a0e438f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj"
Oct 29 22:09:45.854834 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:45.854689    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60d52b51-0159-402d-8d66-3c453a0e438f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj\" (UID: \"60d52b51-0159-402d-8d66-3c453a0e438f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj"
Oct 29 22:09:45.855384 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:45.855358    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60d52b51-0159-402d-8d66-3c453a0e438f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj\" (UID: \"60d52b51-0159-402d-8d66-3c453a0e438f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj"
Oct 29 22:09:45.907365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:45.907333    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l96hh\" (UniqueName: \"kubernetes.io/projected/60d52b51-0159-402d-8d66-3c453a0e438f-kube-api-access-l96hh\") pod \"payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj\" (UID: \"60d52b51-0159-402d-8d66-3c453a0e438f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj"
Oct 29 22:09:45.931932 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:45.931903    4962 generic.go:356] "Generic (PLEG): container finished" podID="cbd6251b-df2f-4739-ae5d-8adbd277e52c" containerID="dfc0898b2a96c3f21ca0edfc46c187b4a86426ecfa4fd8e31e709aab8893bdea" exitCode=0
Oct 29 22:09:45.932084 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:45.931975    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b" event={"ID":"cbd6251b-df2f-4739-ae5d-8adbd277e52c","Type":"ContainerDied","Data":"dfc0898b2a96c3f21ca0edfc46c187b4a86426ecfa4fd8e31e709aab8893bdea"}
Oct 29 22:09:45.932595 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:45.932573    4962 scope.go:122] "RemoveContainer" containerID="dfc0898b2a96c3f21ca0edfc46c187b4a86426ecfa4fd8e31e709aab8893bdea"
Oct 29 22:09:45.934596 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:45.934569    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:09:45.934696 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:45.934637    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:09:45.934696 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:45.934672    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:09:46.025663 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:46.025632    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj"
Oct 29 22:09:46.601486 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:46.595913    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg"
Oct 29 22:09:46.902172 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:09:46.901351    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d52b51_0159_402d_8d66_3c453a0e438f.slice/crio-ae222378238f8e33269d0e6a32344f6b5aa1e5231314ea9f93cf9ac8e60a78e8 WatchSource:0}: Error finding container ae222378238f8e33269d0e6a32344f6b5aa1e5231314ea9f93cf9ac8e60a78e8: Status 404 returned error can't find the container with id ae222378238f8e33269d0e6a32344f6b5aa1e5231314ea9f93cf9ac8e60a78e8
Oct 29 22:09:46.915581 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:46.914796    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj"]
Oct 29 22:09:46.954185 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:46.954159    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj" event={"ID":"60d52b51-0159-402d-8d66-3c453a0e438f","Type":"ContainerStarted","Data":"ae222378238f8e33269d0e6a32344f6b5aa1e5231314ea9f93cf9ac8e60a78e8"}
Oct 29 22:09:46.956633 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:46.956613    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b" event={"ID":"cbd6251b-df2f-4739-ae5d-8adbd277e52c","Type":"ContainerStarted","Data":"d738693af60f9ea4796875bb8a7eb853ab414ab98c8fe2bb56c17a1735111a63"}
Oct 29 22:09:47.801425 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:47.801389    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch"
Oct 29 22:09:47.816999 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:47.816960    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk"]
Oct 29 22:09:47.817408 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:47.817362    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk" podUID="1da3b722-7a00-41ea-969d-d1cfe43314a9" containerName="maas-api" containerID="cri-o://514236a2995511da9e85d53f96a84f2ac36b305130452b862a2a752e8f84ad97" gracePeriod=30
Oct 29 22:09:47.840599 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:47.840567    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"]
Oct 29 22:09:47.854781 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:47.853783    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:47.854918 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:47.854902    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:09:47.859305 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:47.859266    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 29 22:09:47.859487 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:47.859460    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b" podUID="cbd6251b-df2f-4739-ae5d-8adbd277e52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 29 22:09:47.869306 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:47.869270    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-df42c7-openshift-default-dockercfg-jqslj\""
Oct 29 22:09:47.870425 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:47.870399    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"]
Oct 29 22:09:47.963692 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:47.963657    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1693a089-535d-4f1d-815b-075639d9b9ab-istio-token\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:47.964086 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:47.963703    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1693a089-535d-4f1d-815b-075639d9b9ab-istiod-ca-cert\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:47.964086 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:47.963820    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1693a089-535d-4f1d-815b-075639d9b9ab-istio-podinfo\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:47.964086 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:47.963853    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-istio-envoy\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:47.964086 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:47.963884    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-workload-certs\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:47.964086 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:47.964034    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-credential-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:47.964334 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:47.964108    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx67k\" (UniqueName: \"kubernetes.io/projected/1693a089-535d-4f1d-815b-075639d9b9ab-kube-api-access-tx67k\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:47.964334 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:47.964227    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-workload-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:47.964334 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:47.964280    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-istio-data\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:47.967490 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:47.967103    4962 generic.go:356] "Generic (PLEG): container finished" podID="1da3b722-7a00-41ea-969d-d1cfe43314a9" containerID="514236a2995511da9e85d53f96a84f2ac36b305130452b862a2a752e8f84ad97" exitCode=0
Oct 29 22:09:47.967490 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:47.967182    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk" event={"ID":"1da3b722-7a00-41ea-969d-d1cfe43314a9","Type":"ContainerDied","Data":"514236a2995511da9e85d53f96a84f2ac36b305130452b862a2a752e8f84ad97"}
Oct 29 22:09:47.969969 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:47.969949    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj" event={"ID":"60d52b51-0159-402d-8d66-3c453a0e438f","Type":"ContainerStarted","Data":"f67a8d7807ac3e5b57cdaf1a01e11957d06cc25f1288bf19b41214797935c5cc"}
Oct 29 22:09:47.970141 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:47.970124    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj"
Oct 29 22:09:48.027343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.027261    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj" podStartSLOduration=3.027246497 podStartE2EDuration="3.027246497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:48.023862038 +0000 UTC m=+1409.096819017" watchObservedRunningTime="2026-10-29 22:09:48.027246497 +0000 UTC m=+1409.100203460"
Oct 29 22:09:48.066363 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.066235    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1693a089-535d-4f1d-815b-075639d9b9ab-istio-token\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:48.066771 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.066277    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1693a089-535d-4f1d-815b-075639d9b9ab-istiod-ca-cert\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:48.067231 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.066960    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1693a089-535d-4f1d-815b-075639d9b9ab-istio-podinfo\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:48.067231 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.067006    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-istio-envoy\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:48.067231 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.067042    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-workload-certs\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:48.068199 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.067603    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-workload-certs\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:48.068199 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.067646    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-credential-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:48.068199 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.067794    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx67k\" (UniqueName: \"kubernetes.io/projected/1693a089-535d-4f1d-815b-075639d9b9ab-kube-api-access-tx67k\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:48.068199 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.067952    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-workload-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:48.068199 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.067999    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-istio-data\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:48.069648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.068539    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-workload-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:48.069648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.068809    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-istio-data\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:48.069648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.068872    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-credential-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:48.070228 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.070209    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1693a089-535d-4f1d-815b-075639d9b9ab-istiod-ca-cert\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:48.070662 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.070642    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-istio-envoy\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:48.071214 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.071192    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1693a089-535d-4f1d-815b-075639d9b9ab-istio-podinfo\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:48.129065 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.121163    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1693a089-535d-4f1d-815b-075639d9b9ab-istio-token\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:48.129065 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.123087    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx67k\" (UniqueName: \"kubernetes.io/projected/1693a089-535d-4f1d-815b-075639d9b9ab-kube-api-access-tx67k\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:48.218477 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.218446    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk"
Oct 29 22:09:48.231568 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.231538    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:48.305313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.286450    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1da3b722-7a00-41ea-969d-d1cfe43314a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1da3b722-7a00-41ea-969d-d1cfe43314a9-maas-api-tls\") pod \"1da3b722-7a00-41ea-969d-d1cfe43314a9\" (UID: \"1da3b722-7a00-41ea-969d-d1cfe43314a9\") "
Oct 29 22:09:48.305313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.286557    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1da3b722-7a00-41ea-969d-d1cfe43314a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da3b722-7a00-41ea-969d-d1cfe43314a9-openshift-service-ca\") pod \"1da3b722-7a00-41ea-969d-d1cfe43314a9\" (UID: \"1da3b722-7a00-41ea-969d-d1cfe43314a9\") "
Oct 29 22:09:48.305313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.286752    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1da3b722-7a00-41ea-969d-d1cfe43314a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da3b722-7a00-41ea-969d-d1cfe43314a9-metrics-certs\") pod \"1da3b722-7a00-41ea-969d-d1cfe43314a9\" (UID: \"1da3b722-7a00-41ea-969d-d1cfe43314a9\") "
Oct 29 22:09:48.305313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.287021    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1da3b722-7a00-41ea-969d-d1cfe43314a9-kube-api-access-nqs6t\" (UniqueName: \"kubernetes.io/projected/1da3b722-7a00-41ea-969d-d1cfe43314a9-kube-api-access-nqs6t\") pod \"1da3b722-7a00-41ea-969d-d1cfe43314a9\" (UID: \"1da3b722-7a00-41ea-969d-d1cfe43314a9\") "
Oct 29 22:09:48.306064 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.305806    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1da3b722-7a00-41ea-969d-d1cfe43314a9-maas-api-tls" pod "1da3b722-7a00-41ea-969d-d1cfe43314a9" (UID: "1da3b722-7a00-41ea-969d-d1cfe43314a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:09:48.306372 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.306348    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da3b722-7a00-41ea-969d-d1cfe43314a9-openshift-service-ca" pod "1da3b722-7a00-41ea-969d-d1cfe43314a9" (UID: "1da3b722-7a00-41ea-969d-d1cfe43314a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:09:48.308950 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.308925    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1da3b722-7a00-41ea-969d-d1cfe43314a9-metrics-certs" pod "1da3b722-7a00-41ea-969d-d1cfe43314a9" (UID: "1da3b722-7a00-41ea-969d-d1cfe43314a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:09:48.309832 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.309794    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da3b722-7a00-41ea-969d-d1cfe43314a9-kube-api-access-nqs6t" pod "1da3b722-7a00-41ea-969d-d1cfe43314a9" (UID: "1da3b722-7a00-41ea-969d-d1cfe43314a9"). InnerVolumeSpecName "kube-api-access-nqs6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:09:48.400140 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.399997    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqs6t\" (UniqueName: \"kubernetes.io/projected/1da3b722-7a00-41ea-969d-d1cfe43314a9-kube-api-access-nqs6t\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:09:48.400140 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.400033    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1da3b722-7a00-41ea-969d-d1cfe43314a9-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:09:48.400140 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.400047    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1da3b722-7a00-41ea-969d-d1cfe43314a9-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:09:48.400140 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.400060    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1da3b722-7a00-41ea-969d-d1cfe43314a9-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:09:48.855250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.855208    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"]
Oct 29 22:09:48.855926 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.855894    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 29 22:09:48.856143 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.856118    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b" podUID="cbd6251b-df2f-4739-ae5d-8adbd277e52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 29 22:09:48.861772 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:48.861744    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5"
Oct 29 22:09:49.118258 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:09:49.117439    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1693a089_535d_4f1d_815b_075639d9b9ab.slice/crio-c66286813b5eaf4da8ac96a43aeaf9e4f7ca5215c485cd16d085e14f75c64621 WatchSource:0}: Error finding container c66286813b5eaf4da8ac96a43aeaf9e4f7ca5215c485cd16d085e14f75c64621: Status 404 returned error can't find the container with id c66286813b5eaf4da8ac96a43aeaf9e4f7ca5215c485cd16d085e14f75c64621
Oct 29 22:09:49.165449 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.164569    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:09:49.165449 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.164670    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:09:49.165449 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.164717    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:09:49.238950 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.235775    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk"
Oct 29 22:09:49.238950 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.237131    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"]
Oct 29 22:09:49.238950 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.237967    4962 cpu_manager.go:414] "containerMap: removing container" podUID="1da3b722-7a00-41ea-969d-d1cfe43314a9" containerName="maas-api"
Oct 29 22:09:49.238950 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.237981    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1da3b722-7a00-41ea-969d-d1cfe43314a9" containerName="maas-api"
Oct 29 22:09:49.238950 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.238112    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="1da3b722-7a00-41ea-969d-d1cfe43314a9" containerName="maas-api"
Oct 29 22:09:49.244039 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.244013    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"]
Oct 29 22:09:49.244161 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.244059    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk" event={"ID":"1da3b722-7a00-41ea-969d-d1cfe43314a9","Type":"ContainerDied","Data":"2176cb11885640f8e60aefbf0f21c9639544b65fb83505c5dad0becfecba7bfa"}
Oct 29 22:09:49.244161 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.244108    4962 scope.go:122] "RemoveContainer" containerID="514236a2995511da9e85d53f96a84f2ac36b305130452b862a2a752e8f84ad97"
Oct 29 22:09:49.244382 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.244359    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.263309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.262630    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-edc2d3-openshift-default-dockercfg-2pkvj\""
Oct 29 22:09:49.546881 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.546844    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.547028 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.546942    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzrl9\" (UniqueName: \"kubernetes.io/projected/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-kube-api-access-jzrl9\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.547028 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.547009    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.547126 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.547085    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.547236 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.547217    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.547442 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.547359    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.547442 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.547390    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.547548 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.547443    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.547607 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.547552    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.607736 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.607331    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk"]
Oct 29 22:09:49.612191 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.612162    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-dd38c2-847b9695c8-qlnvk"]
Oct 29 22:09:49.649458 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.649377    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.649617 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.649523    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.649688 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.649653    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.649755 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.649687    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.649755 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.649741    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.649868 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.649848    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.649963 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.649943    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.650102 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.650028    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzrl9\" (UniqueName: \"kubernetes.io/projected/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-kube-api-access-jzrl9\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.650102 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.650096    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.650257 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.650230    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.650581 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.650554    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.650723 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.650618    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.650946 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.650925    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.651544 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.651524    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.653022 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.653003    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.658049 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.657707    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.695157 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.695128    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzrl9\" (UniqueName: \"kubernetes.io/projected/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-kube-api-access-jzrl9\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.708234 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.708189    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:49.852567 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.852525    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 29 22:09:49.852740 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.852588    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b" podUID="cbd6251b-df2f-4739-ae5d-8adbd277e52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 29 22:09:49.855764 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:49.855734    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:50.247670 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:50.247573    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs" event={"ID":"1693a089-535d-4f1d-815b-075639d9b9ab","Type":"ContainerStarted","Data":"0c2aff4fe1efcef4e14fea0b1742337c09c48cecb057ca7936e78554cb7434ad"}
Oct 29 22:09:50.247670 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:50.247656    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs" event={"ID":"1693a089-535d-4f1d-815b-075639d9b9ab","Type":"ContainerStarted","Data":"c66286813b5eaf4da8ac96a43aeaf9e4f7ca5215c485cd16d085e14f75c64621"}
Oct 29 22:09:50.305052 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:50.304991    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs" podStartSLOduration=3.304972069 podStartE2EDuration="3.304972069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:50.297999892 +0000 UTC m=+1411.370956878" watchObservedRunningTime="2026-10-29 22:09:50.304972069 +0000 UTC m=+1411.377929034"
Oct 29 22:09:50.435330 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:50.432281    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"]
Oct 29 22:09:50.437743 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:09:50.437711    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c84e6b4_e8f7_49b6_ab65_7f677e6fe28a.slice/crio-752897f914d6e398bff79963222fb6afc0d098a0dee026aeb18855500db817de WatchSource:0}: Error finding container 752897f914d6e398bff79963222fb6afc0d098a0dee026aeb18855500db817de: Status 404 returned error can't find the container with id 752897f914d6e398bff79963222fb6afc0d098a0dee026aeb18855500db817de
Oct 29 22:09:50.441452 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:50.441415    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:09:50.441538 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:50.441500    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:09:50.441596 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:50.441579    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:09:50.651549 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:50.651509    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:09:50.858136 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:50.857680    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 29 22:09:50.858136 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:50.857736    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b" podUID="cbd6251b-df2f-4739-ae5d-8adbd277e52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 29 22:09:51.275412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:51.275372    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 29 22:09:51.276016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:51.275985    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs" podUID="1693a089-535d-4f1d-815b-075639d9b9ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 29 22:09:51.347543 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:51.347511    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1da3b722-7a00-41ea-969d-d1cfe43314a9" path="/var/lib/kubelet/pods/1da3b722-7a00-41ea-969d-d1cfe43314a9/volumes"
Oct 29 22:09:51.348258 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:51.348235    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j" event={"ID":"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a","Type":"ContainerStarted","Data":"e817669945643f3f5e7efa4e1c816763c2c14aa7338c7fca3182566de1cad3ac"}
Oct 29 22:09:51.348433 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:51.348416    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j" event={"ID":"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a","Type":"ContainerStarted","Data":"752897f914d6e398bff79963222fb6afc0d098a0dee026aeb18855500db817de"}
Oct 29 22:09:51.348533 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:51.348522    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:09:51.366987 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:51.365690    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9"]
Oct 29 22:09:51.421382 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:51.421348    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b"]
Oct 29 22:09:51.431642 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:51.426313    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j" podStartSLOduration=3.426279847 podStartE2EDuration="3.426279847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:09:51.409048692 +0000 UTC m=+1412.482005657" watchObservedRunningTime="2026-10-29 22:09:51.426279847 +0000 UTC m=+1412.499236826"
Oct 29 22:09:51.885161 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:51.882112    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:09:51.885161 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:51.883745    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 29 22:09:51.885161 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:51.883798    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b" podUID="cbd6251b-df2f-4739-ae5d-8adbd277e52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 29 22:09:51.910694 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:51.904391    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 29 22:09:51.910694 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:51.904454    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j" podUID="4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 29 22:09:52.240184 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:52.240143    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 29 22:09:52.240496 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:52.240465    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs" podUID="1693a089-535d-4f1d-815b-075639d9b9ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 29 22:09:52.852752 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:52.852706    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 29 22:09:52.853307 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:52.852775    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b" podUID="cbd6251b-df2f-4739-ae5d-8adbd277e52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 29 22:09:52.856437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:52.856407    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 29 22:09:52.856642 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:52.856619    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j" podUID="4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 29 22:09:53.235312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:53.235083    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 29 22:09:53.235312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:53.235140    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs" podUID="1693a089-535d-4f1d-815b-075639d9b9ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 29 22:09:53.858325 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:53.856719    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body=
Oct 29 22:09:53.858325 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:53.856782    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b" podUID="cbd6251b-df2f-4739-ae5d-8adbd277e52c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused"
Oct 29 22:09:53.858908 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:53.858823    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 29 22:09:53.858908 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:53.858889    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j" podUID="4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 29 22:09:54.251331 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:54.239020    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 29 22:09:54.251331 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:54.239081    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs" podUID="1693a089-535d-4f1d-815b-075639d9b9ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 29 22:09:54.395632 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:54.395595    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-d4fgq_3472b532-36f0-4a0d-9d04-27ec9a9f32e6/payload-processing/0.log"
Oct 29 22:09:54.626636 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:54.626458    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-d4fgq_3472b532-36f0-4a0d-9d04-27ec9a9f32e6/payload-processing/0.log"
Oct 29 22:09:54.856675 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:54.856638    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 29 22:09:54.857389 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:54.857353    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j" podUID="4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 29 22:09:55.063027 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:55.062984    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:09:55.065102 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:55.065079    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:09:55.237048 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:55.237006    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 29 22:09:55.237246 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:55.237065    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs" podUID="1693a089-535d-4f1d-815b-075639d9b9ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 29 22:09:55.860823 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:55.860773    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 29 22:09:55.884475 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:55.884430    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j" podUID="4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 29 22:09:56.232217 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:56.232180    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 29 22:09:56.232662 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:56.232243    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs" podUID="1693a089-535d-4f1d-815b-075639d9b9ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 29 22:09:56.366252 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:56.366181    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9" podUID="3e9d173f-c37f-4675-9ffe-9ca09e25794c" containerName="payload-processing" containerID="cri-o://ae5cea13586587b5e228bdfbcdd1a3f0ac685c42ac83c5ef467b748a742bd1c1" gracePeriod=25
Oct 29 22:09:56.432387 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:56.432344    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b" podUID="528077a0-2c34-4ff3-b229-3af6a3cebb16" containerName="payload-pre-processing" containerID="cri-o://1f72ba6c609487567e1713ad1b605b67e272531b867e565c4abd03bf0b76ddd5" gracePeriod=25
Oct 29 22:09:56.857200 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:56.856639    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 29 22:09:56.857200 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:56.856703    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j" podUID="4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 29 22:09:57.013941 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:57.011998    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9"
Oct 29 22:09:57.118394 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:57.114922    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e9d173f-c37f-4675-9ffe-9ca09e25794c-kube-api-access-5fd7x\" (UniqueName: \"kubernetes.io/projected/3e9d173f-c37f-4675-9ffe-9ca09e25794c-kube-api-access-5fd7x\") pod \"3e9d173f-c37f-4675-9ffe-9ca09e25794c\" (UID: \"3e9d173f-c37f-4675-9ffe-9ca09e25794c\") "
Oct 29 22:09:57.118394 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:57.115175    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e9d173f-c37f-4675-9ffe-9ca09e25794c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9d173f-c37f-4675-9ffe-9ca09e25794c-plugins-config-volume\") pod \"3e9d173f-c37f-4675-9ffe-9ca09e25794c\" (UID: \"3e9d173f-c37f-4675-9ffe-9ca09e25794c\") "
Oct 29 22:09:57.223517 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:57.198113    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9d173f-c37f-4675-9ffe-9ca09e25794c-plugins-config-volume" pod "3e9d173f-c37f-4675-9ffe-9ca09e25794c" (UID: "3e9d173f-c37f-4675-9ffe-9ca09e25794c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:09:57.223517 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:57.214904    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9d173f-c37f-4675-9ffe-9ca09e25794c-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:09:57.418410 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:57.383800    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 29 22:09:57.445693 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:57.445645    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs" podUID="1693a089-535d-4f1d-815b-075639d9b9ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 29 22:09:57.475321 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:57.427005    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e9d173f-c37f-4675-9ffe-9ca09e25794c-kube-api-access-5fd7x" pod "3e9d173f-c37f-4675-9ffe-9ca09e25794c" (UID: "3e9d173f-c37f-4675-9ffe-9ca09e25794c"). InnerVolumeSpecName "kube-api-access-5fd7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:09:57.511371 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:57.511313    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fd7x\" (UniqueName: \"kubernetes.io/projected/3e9d173f-c37f-4675-9ffe-9ca09e25794c-kube-api-access-5fd7x\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:09:57.541310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:57.541231    4962 generic.go:356] "Generic (PLEG): container finished" podID="528077a0-2c34-4ff3-b229-3af6a3cebb16" containerID="1f72ba6c609487567e1713ad1b605b67e272531b867e565c4abd03bf0b76ddd5" exitCode=0
Oct 29 22:09:57.602333 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:57.600012    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b" event={"ID":"528077a0-2c34-4ff3-b229-3af6a3cebb16","Type":"ContainerDied","Data":"1f72ba6c609487567e1713ad1b605b67e272531b867e565c4abd03bf0b76ddd5"}
Oct 29 22:09:57.802475 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:57.653313    4962 generic.go:356] "Generic (PLEG): container finished" podID="3e9d173f-c37f-4675-9ffe-9ca09e25794c" containerID="ae5cea13586587b5e228bdfbcdd1a3f0ac685c42ac83c5ef467b748a742bd1c1" exitCode=0
Oct 29 22:09:57.802475 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:57.653668    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9" event={"ID":"3e9d173f-c37f-4675-9ffe-9ca09e25794c","Type":"ContainerDied","Data":"ae5cea13586587b5e228bdfbcdd1a3f0ac685c42ac83c5ef467b748a742bd1c1"}
Oct 29 22:09:57.802475 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:57.653731    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9" event={"ID":"3e9d173f-c37f-4675-9ffe-9ca09e25794c","Type":"ContainerDied","Data":"53fe6fbcd77466181dd48af8249b0fc249aa86d247fa04022f84d23ae45efc61"}
Oct 29 22:09:57.802475 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:57.653766    4962 scope.go:122] "RemoveContainer" containerID="ae5cea13586587b5e228bdfbcdd1a3f0ac685c42ac83c5ef467b748a742bd1c1"
Oct 29 22:09:57.802475 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:57.653960    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9"
Oct 29 22:09:57.887310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:57.878574    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"]
Oct 29 22:09:57.888663 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:57.888618    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b" podUID="cbd6251b-df2f-4739-ae5d-8adbd277e52c" containerName="istio-proxy" containerID="cri-o://d738693af60f9ea4796875bb8a7eb853ab414ab98c8fe2bb56c17a1735111a63" gracePeriod=30
Oct 29 22:09:57.898484 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:57.889183    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:57.987342 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:57.983195    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:58.009326 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.004621    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:58.009730 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.009694    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 29 22:09:58.009853 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.009759    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j" podUID="4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 29 22:09:58.113360 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.107040    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:58.145162 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.145126    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"]
Oct 29 22:09:58.145663 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.145628    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb" podUID="977f25cb-ac5d-4290-a9a1-59322cc86f3d" containerName="istio-proxy" containerID="cri-o://1fa8cbaced73613c14dc2976e215755a76950eeb60e721b1279ce533213e469d" gracePeriod=30
Oct 29 22:09:58.285083 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.185392    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:58.380410 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.380282    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b"
Oct 29 22:09:58.411791 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.411747    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 29 22:09:58.412109 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.412050    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs" podUID="1693a089-535d-4f1d-815b-075639d9b9ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 29 22:09:58.471044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.471001    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:58.496866 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.495187    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:09:58.514120 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.514084    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528077a0-2c34-4ff3-b229-3af6a3cebb16-kube-api-access-p6cgj\" (UniqueName: \"kubernetes.io/projected/528077a0-2c34-4ff3-b229-3af6a3cebb16-kube-api-access-p6cgj\") pod \"528077a0-2c34-4ff3-b229-3af6a3cebb16\" (UID: \"528077a0-2c34-4ff3-b229-3af6a3cebb16\") "
Oct 29 22:09:58.539730 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.539689    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:58.543689 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.543661    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/528077a0-2c34-4ff3-b229-3af6a3cebb16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/528077a0-2c34-4ff3-b229-3af6a3cebb16-plugins-config-volume\") pod \"528077a0-2c34-4ff3-b229-3af6a3cebb16\" (UID: \"528077a0-2c34-4ff3-b229-3af6a3cebb16\") "
Oct 29 22:09:58.564685 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.564644    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528077a0-2c34-4ff3-b229-3af6a3cebb16-plugins-config-volume" pod "528077a0-2c34-4ff3-b229-3af6a3cebb16" (UID: "528077a0-2c34-4ff3-b229-3af6a3cebb16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:09:58.568572 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.568540    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"]
Oct 29 22:09:58.592432 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.592383    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d" podUID="b0573e45-68c1-4509-abae-91401555180e" containerName="istio-proxy" containerID="cri-o://48a68d1dee50f13e4f72a6d12b111253b589e6af1052ac6ace6e4e22d806a140" gracePeriod=30
Oct 29 22:09:58.592782 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.589097    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:58.744330 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.731840    4962 scope.go:122] "RemoveContainer" containerID="ae5cea13586587b5e228bdfbcdd1a3f0ac685c42ac83c5ef467b748a742bd1c1"
Oct 29 22:09:58.744330 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.732029    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b" podUID="528077a0-2c34-4ff3-b229-3af6a3cebb16" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.146:50052\" within 1s: context deadline exceeded"
Oct 29 22:09:58.744330 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:09:58.732168    4962 logging.go:55] [core] [Channel #3035 SubChannel #3036]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.146:50052", ServerName: "10.132.0.146:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.146:50052: operation was canceled"
Oct 29 22:09:58.746259 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.746218    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/528077a0-2c34-4ff3-b229-3af6a3cebb16-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:09:58.755936 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:09:58.755883    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae5cea13586587b5e228bdfbcdd1a3f0ac685c42ac83c5ef467b748a742bd1c1\": container with ID starting with ae5cea13586587b5e228bdfbcdd1a3f0ac685c42ac83c5ef467b748a742bd1c1 not found: ID does not exist" containerID="ae5cea13586587b5e228bdfbcdd1a3f0ac685c42ac83c5ef467b748a742bd1c1"
Oct 29 22:09:58.756185 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.756153    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae5cea13586587b5e228bdfbcdd1a3f0ac685c42ac83c5ef467b748a742bd1c1"} err="failed to get container status \"ae5cea13586587b5e228bdfbcdd1a3f0ac685c42ac83c5ef467b748a742bd1c1\": rpc error: code = NotFound desc = could not find container \"ae5cea13586587b5e228bdfbcdd1a3f0ac685c42ac83c5ef467b748a742bd1c1\": container with ID starting with ae5cea13586587b5e228bdfbcdd1a3f0ac685c42ac83c5ef467b748a742bd1c1 not found: ID does not exist"
Oct 29 22:09:58.845339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.802315    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:58.907309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.900493    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:58.911977 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.911933    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:58.916736 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.916709    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:09:59.056398 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:58.944214    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 29 22:09:59.056398 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.024415    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j" podUID="4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 29 22:09:59.172592 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.172489    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:59.179657 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.179619    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:09:59.285672 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.202446    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:59.285672 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.281452    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:09:59.290357 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.286661    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 22:09:59.290357 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.286721    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:09:59.311404 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.206415    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:09:59.311404 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.306858    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"]
Oct 29 22:09:59.311404 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.306918    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s"]
Oct 29 22:09:59.311404 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.307101    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s" podUID="4f572782-bbae-4cb0-a055-b4c9d9e8db5a" containerName="maas-api" containerID="cri-o://55c5e0f5c9409b457b573ed94923e01ad880bc0597331faf94a45dd43b747867" gracePeriod=30
Oct 29 22:09:59.311404 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.307946    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf" podUID="a8ab68a7-4c08-494f-95c7-006502508963" containerName="istio-proxy" containerID="cri-o://146a99d6af0771756872912bf2cb97a8bcf63c3025bec77e33ac492b522fd026" gracePeriod=30
Oct 29 22:09:59.336341 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.313874    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 29 22:09:59.336341 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.313919    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs" podUID="1693a089-535d-4f1d-815b-075639d9b9ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 29 22:09:59.336341 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.335495    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9"]
Oct 29 22:09:59.344575 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.343559    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a9e67f-69b9c86ccd-z9wm9"]
Oct 29 22:09:59.352143 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.352112    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528077a0-2c34-4ff3-b229-3af6a3cebb16-kube-api-access-p6cgj" pod "528077a0-2c34-4ff3-b229-3af6a3cebb16" (UID: "528077a0-2c34-4ff3-b229-3af6a3cebb16"). InnerVolumeSpecName "kube-api-access-p6cgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:09:59.395497 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.391749    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6cgj\" (UniqueName: \"kubernetes.io/projected/528077a0-2c34-4ff3-b229-3af6a3cebb16-kube-api-access-p6cgj\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:09:59.426517 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.426486    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b"
Oct 29 22:09:59.465950 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.465419    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e9d173f-c37f-4675-9ffe-9ca09e25794c" path="/var/lib/kubelet/pods/3e9d173f-c37f-4675-9ffe-9ca09e25794c/volumes"
Oct 29 22:09:59.494312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.487627    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:09:59.650388 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.642695    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:09:59.850479 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.850445    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8"]
Oct 29 22:09:59.855506 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.854494    4962 cpu_manager.go:414] "containerMap: removing container" podUID="528077a0-2c34-4ff3-b229-3af6a3cebb16" containerName="payload-pre-processing"
Oct 29 22:09:59.855710 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.855689    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="528077a0-2c34-4ff3-b229-3af6a3cebb16" containerName="payload-pre-processing"
Oct 29 22:09:59.856061 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.856044    4962 cpu_manager.go:414] "containerMap: removing container" podUID="3e9d173f-c37f-4675-9ffe-9ca09e25794c" containerName="payload-processing"
Oct 29 22:09:59.856157 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.856145    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e9d173f-c37f-4675-9ffe-9ca09e25794c" containerName="payload-processing"
Oct 29 22:09:59.857259 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.857237    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="528077a0-2c34-4ff3-b229-3af6a3cebb16" containerName="payload-pre-processing"
Oct 29 22:09:59.857448 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.857432    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e9d173f-c37f-4675-9ffe-9ca09e25794c" containerName="payload-processing"
Oct 29 22:09:59.872517 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.869330    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 29 22:09:59.872517 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.869386    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j" podUID="4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 29 22:09:59.891019 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.890977    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj"
Oct 29 22:09:59.979980 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.979940    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8"
Oct 29 22:10:00.057310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:09:59.891301    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b" event={"ID":"528077a0-2c34-4ff3-b229-3af6a3cebb16","Type":"ContainerDied","Data":"bd13b00614ad4c8eec324d34cfd2deab3af17a96e9abbe66650bbfbd26a86517"}
Oct 29 22:10:00.112623 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.112589    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f82846-serving-cert\""
Oct 29 22:10:00.153625 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.153594    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:10:00.154115 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.154091    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8"]
Oct 29 22:10:00.200766 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.199336    4962 scope.go:122] "RemoveContainer" containerID="1f72ba6c609487567e1713ad1b605b67e272531b867e565c4abd03bf0b76ddd5"
Oct 29 22:10:00.209953 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.209932    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 22:10:00.210494 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.210472    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:10:00.210751 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.210733    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 22:10:00.258726 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.258515    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:00.259724 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.259669    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Oct 29 22:10:00.259902 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.259883    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs" podUID="1693a089-535d-4f1d-815b-075639d9b9ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Oct 29 22:10:00.265266 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.265153    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktk5g\" (UniqueName: \"kubernetes.io/projected/df517504-590f-4f2b-8ab1-5e92e1f1936c-kube-api-access-ktk5g\") pod \"maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8\" (UID: \"df517504-590f-4f2b-8ab1-5e92e1f1936c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8"
Oct 29 22:10:00.265620 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.265588    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df517504-590f-4f2b-8ab1-5e92e1f1936c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8\" (UID: \"df517504-590f-4f2b-8ab1-5e92e1f1936c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8"
Oct 29 22:10:00.266043 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.266024    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df517504-590f-4f2b-8ab1-5e92e1f1936c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8\" (UID: \"df517504-590f-4f2b-8ab1-5e92e1f1936c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8"
Oct 29 22:10:00.324273 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.322984    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:00.332965 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.332910    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df517504-590f-4f2b-8ab1-5e92e1f1936c-metrics-certs\") pod \"maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8\" (UID: \"df517504-590f-4f2b-8ab1-5e92e1f1936c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8"
Oct 29 22:10:00.364047 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.364017    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:10:00.364661 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.364640    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:10:00.368770 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.368751    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:10:00.369168 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.369150    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 22:10:00.527543 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.527516    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b"]
Oct 29 22:10:00.531330 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.531300    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a9e67f-869b689d6c-kz99b"]
Oct 29 22:10:00.533431 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.533385    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df517504-590f-4f2b-8ab1-5e92e1f1936c-metrics-certs\") pod \"maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8\" (UID: \"df517504-590f-4f2b-8ab1-5e92e1f1936c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8"
Oct 29 22:10:00.534224 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.534202    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktk5g\" (UniqueName: \"kubernetes.io/projected/df517504-590f-4f2b-8ab1-5e92e1f1936c-kube-api-access-ktk5g\") pod \"maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8\" (UID: \"df517504-590f-4f2b-8ab1-5e92e1f1936c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8"
Oct 29 22:10:00.534664 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.534645    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df517504-590f-4f2b-8ab1-5e92e1f1936c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8\" (UID: \"df517504-590f-4f2b-8ab1-5e92e1f1936c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8"
Oct 29 22:10:00.535484 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.535466    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df517504-590f-4f2b-8ab1-5e92e1f1936c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8\" (UID: \"df517504-590f-4f2b-8ab1-5e92e1f1936c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8"
Oct 29 22:10:00.538162 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.534240    4962 generic.go:356] "Generic (PLEG): container finished" podID="4f572782-bbae-4cb0-a055-b4c9d9e8db5a" containerID="55c5e0f5c9409b457b573ed94923e01ad880bc0597331faf94a45dd43b747867" exitCode=0
Oct 29 22:10:00.539135 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.534261    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s" event={"ID":"4f572782-bbae-4cb0-a055-b4c9d9e8db5a","Type":"ContainerDied","Data":"55c5e0f5c9409b457b573ed94923e01ad880bc0597331faf94a45dd43b747867"}
Oct 29 22:10:00.540555 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.537984    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df517504-590f-4f2b-8ab1-5e92e1f1936c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8\" (UID: \"df517504-590f-4f2b-8ab1-5e92e1f1936c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8"
Oct 29 22:10:00.669312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.628553    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df517504-590f-4f2b-8ab1-5e92e1f1936c-metrics-certs\") pod \"maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8\" (UID: \"df517504-590f-4f2b-8ab1-5e92e1f1936c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8"
Oct 29 22:10:00.698170 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.675949    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktk5g\" (UniqueName: \"kubernetes.io/projected/df517504-590f-4f2b-8ab1-5e92e1f1936c-kube-api-access-ktk5g\") pod \"maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8\" (UID: \"df517504-590f-4f2b-8ab1-5e92e1f1936c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8"
Oct 29 22:10:00.698942 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.698891    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df517504-590f-4f2b-8ab1-5e92e1f1936c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8\" (UID: \"df517504-590f-4f2b-8ab1-5e92e1f1936c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8"
Oct 29 22:10:00.856242 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.856154    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body=
Oct 29 22:10:00.856516 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.856482    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j" podUID="4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused"
Oct 29 22:10:00.862038 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.862009    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s"
Oct 29 22:10:00.898898 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:10:00.898855    4962 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f572782_bbae_4cb0_a055_b4c9d9e8db5a.slice/crio-55c5e0f5c9409b457b573ed94923e01ad880bc0597331faf94a45dd43b747867.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f572782_bbae_4cb0_a055_b4c9d9e8db5a.slice/crio-conmon-55c5e0f5c9409b457b573ed94923e01ad880bc0597331faf94a45dd43b747867.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:10:00.930819 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:00.930789    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8"
Oct 29 22:10:01.011688 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.010305    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-openshift-service-ca\") pod \"4f572782-bbae-4cb0-a055-b4c9d9e8db5a\" (UID: \"4f572782-bbae-4cb0-a055-b4c9d9e8db5a\") "
Oct 29 22:10:01.011688 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.010365    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-maas-api-tls\") pod \"4f572782-bbae-4cb0-a055-b4c9d9e8db5a\" (UID: \"4f572782-bbae-4cb0-a055-b4c9d9e8db5a\") "
Oct 29 22:10:01.011688 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.010475    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-kube-api-access-d7wm5\" (UniqueName: \"kubernetes.io/projected/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-kube-api-access-d7wm5\") pod \"4f572782-bbae-4cb0-a055-b4c9d9e8db5a\" (UID: \"4f572782-bbae-4cb0-a055-b4c9d9e8db5a\") "
Oct 29 22:10:01.011688 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.010681    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-metrics-certs\") pod \"4f572782-bbae-4cb0-a055-b4c9d9e8db5a\" (UID: \"4f572782-bbae-4cb0-a055-b4c9d9e8db5a\") "
Oct 29 22:10:01.013637 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.013596    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-openshift-service-ca" pod "4f572782-bbae-4cb0-a055-b4c9d9e8db5a" (UID: "4f572782-bbae-4cb0-a055-b4c9d9e8db5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:01.015905 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.015862    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-metrics-certs" pod "4f572782-bbae-4cb0-a055-b4c9d9e8db5a" (UID: "4f572782-bbae-4cb0-a055-b4c9d9e8db5a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:10:01.017344 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.017312    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-maas-api-tls" pod "4f572782-bbae-4cb0-a055-b4c9d9e8db5a" (UID: "4f572782-bbae-4cb0-a055-b4c9d9e8db5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:10:01.032204 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.032173    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-kube-api-access-d7wm5" pod "4f572782-bbae-4cb0-a055-b4c9d9e8db5a" (UID: "4f572782-bbae-4cb0-a055-b4c9d9e8db5a"). InnerVolumeSpecName "kube-api-access-d7wm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:01.065940 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.065908    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="528077a0-2c34-4ff3-b229-3af6a3cebb16" path="/var/lib/kubelet/pods/528077a0-2c34-4ff3-b229-3af6a3cebb16/volumes"
Oct 29 22:10:01.113573 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.113235    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:01.113573 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.113269    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:01.113573 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.113300    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7wm5\" (UniqueName: \"kubernetes.io/projected/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-kube-api-access-d7wm5\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:01.113573 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.113335    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f572782-bbae-4cb0-a055-b4c9d9e8db5a-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:01.237159 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.237125    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:10:01.237496 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.237472    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:10:01.238225 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.238208    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:10:01.269727 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:10:01.269696    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf517504_590f_4f2b_8ab1_5e92e1f1936c.slice/crio-af61d17f1c3d952876a4223053d9738f653f3b76b76eec92da0e78644e20373d WatchSource:0}: Error finding container af61d17f1c3d952876a4223053d9738f653f3b76b76eec92da0e78644e20373d: Status 404 returned error can't find the container with id af61d17f1c3d952876a4223053d9738f653f3b76b76eec92da0e78644e20373d
Oct 29 22:10:01.274274 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.274253    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8"]
Oct 29 22:10:01.560080 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.560053    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s"
Oct 29 22:10:01.560080 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.560068    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s" event={"ID":"4f572782-bbae-4cb0-a055-b4c9d9e8db5a","Type":"ContainerDied","Data":"2c301278bf93dc0a66d2e79a73e557d7e078169a2a050abd6ff3b6bff4f72fd3"}
Oct 29 22:10:01.560550 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.560159    4962 scope.go:122] "RemoveContainer" containerID="55c5e0f5c9409b457b573ed94923e01ad880bc0597331faf94a45dd43b747867"
Oct 29 22:10:01.561768 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.561742    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8" event={"ID":"df517504-590f-4f2b-8ab1-5e92e1f1936c","Type":"ContainerStarted","Data":"af61d17f1c3d952876a4223053d9738f653f3b76b76eec92da0e78644e20373d"}
Oct 29 22:10:01.613273 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.613247    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s"]
Oct 29 22:10:01.618194 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.618168    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-301fcc-64457fbd84-xzx7s"]
Oct 29 22:10:01.860905 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.860867    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:10:01.861425 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.861407    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:10:01.862128 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:01.862109    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:10:02.143869 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.143786    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"]
Oct 29 22:10:02.144703 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.144668    4962 cpu_manager.go:414] "containerMap: removing container" podUID="4f572782-bbae-4cb0-a055-b4c9d9e8db5a" containerName="maas-api"
Oct 29 22:10:02.144703 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.144702    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f572782-bbae-4cb0-a055-b4c9d9e8db5a" containerName="maas-api"
Oct 29 22:10:02.144986 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.144962    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f572782-bbae-4cb0-a055-b4c9d9e8db5a" containerName="maas-api"
Oct 29 22:10:02.148945 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.148925    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"
Oct 29 22:10:02.200348 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.200282    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-b80d5b-serving-cert\""
Oct 29 22:10:02.202105 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.202077    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"]
Oct 29 22:10:02.232230 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.232191    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866\" (UID: \"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"
Oct 29 22:10:02.232392 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.232353    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6x4g\" (UniqueName: \"kubernetes.io/projected/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-kube-api-access-k6x4g\") pod \"maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866\" (UID: \"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"
Oct 29 22:10:02.232576 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.232555    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866\" (UID: \"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"
Oct 29 22:10:02.232669 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.232607    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-metrics-certs\") pod \"maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866\" (UID: \"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"
Oct 29 22:10:02.331009 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.330974    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:10:02.331146 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.331024    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf" podUID="a8ab68a7-4c08-494f-95c7-006502508963" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:10:02.334012 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.333989    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866\" (UID: \"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"
Oct 29 22:10:02.334102 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.334035    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-metrics-certs\") pod \"maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866\" (UID: \"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"
Oct 29 22:10:02.334270 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.334252    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866\" (UID: \"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"
Oct 29 22:10:02.334373 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.334357    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6x4g\" (UniqueName: \"kubernetes.io/projected/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-kube-api-access-k6x4g\") pod \"maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866\" (UID: \"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"
Oct 29 22:10:02.334436 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:10:02.334395    4962 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-serving-cert: secret "maas-api-e2e-ipp-a-b80d5b-serving-cert" not found
Oct 29 22:10:02.334488 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:10:02.334465    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-maas-api-tls podName:ad28e0d7-1aaa-44b8-a284-d3383d08e5cb nodeName:}" failed. No retries permitted until 2026-10-29 22:10:02.834445083 +0000 UTC m=+1423.907402039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-maas-api-tls") pod "maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866" (UID: "ad28e0d7-1aaa-44b8-a284-d3383d08e5cb") : secret "maas-api-e2e-ipp-a-b80d5b-serving-cert" not found
Oct 29 22:10:02.334791 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.334763    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866\" (UID: \"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"
Oct 29 22:10:02.336660 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.336644    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-metrics-certs\") pod \"maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866\" (UID: \"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"
Oct 29 22:10:02.385813 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.385460    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6x4g\" (UniqueName: \"kubernetes.io/projected/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-kube-api-access-k6x4g\") pod \"maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866\" (UID: \"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"
Oct 29 22:10:02.575107 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.575073    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8" event={"ID":"df517504-590f-4f2b-8ab1-5e92e1f1936c","Type":"ContainerStarted","Data":"e3e683c9d02b3999bd45242baddc3f1ffe93d5216cd33fd4582a71edd8c04302"}
Oct 29 22:10:02.575107 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.575185    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8"
Oct 29 22:10:02.619740 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.619694    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8" podStartSLOduration=4.221333543 podStartE2EDuration="4.61968016s" totalImagesPullingTime="398.346617ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:02.617847245 +0000 UTC m=+1423.690804209" watchObservedRunningTime="2026-10-29 22:10:02.61968016 +0000 UTC m=+1423.692637125"
Oct 29 22:10:02.867215 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.867111    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866\" (UID: \"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"
Oct 29 22:10:02.900353 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:02.878259    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866\" (UID: \"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"
Oct 29 22:10:03.082701 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.081016    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"]
Oct 29 22:10:03.087419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.087393    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"
Oct 29 22:10:03.090370 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.089053    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"]
Oct 29 22:10:03.093399 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.092590    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.114847 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.114798    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f572782-bbae-4cb0-a055-b4c9d9e8db5a" path="/var/lib/kubelet/pods/4f572782-bbae-4cb0-a055-b4c9d9e8db5a/volumes"
Oct 29 22:10:03.120558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.120496    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58653d9-6bc5-44c4-97ed-068232825e36-istiod-ca-cert\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.120681 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.120650    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-credential-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.121551 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.120740    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xncl6\" (UniqueName: \"kubernetes.io/projected/f58653d9-6bc5-44c4-97ed-068232825e36-kube-api-access-xncl6\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.121551 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.120803    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-workload-certs\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.121551 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.120868    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-workload-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.121551 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.120905    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-istio-envoy\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.121551 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.120929    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58653d9-6bc5-44c4-97ed-068232825e36-istio-podinfo\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.121551 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.120957    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58653d9-6bc5-44c4-97ed-068232825e36-istio-token\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.121551 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.121283    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-istio-data\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.231760 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.224036    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-istio-data\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.231760 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.224138    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58653d9-6bc5-44c4-97ed-068232825e36-istiod-ca-cert\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.231760 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.224221    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-credential-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.231760 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.224315    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xncl6\" (UniqueName: \"kubernetes.io/projected/f58653d9-6bc5-44c4-97ed-068232825e36-kube-api-access-xncl6\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.231760 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.224379    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-workload-certs\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.231760 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.224446    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-workload-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.231760 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.224472    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-istio-envoy\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.231760 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.224496    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58653d9-6bc5-44c4-97ed-068232825e36-istio-podinfo\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.231760 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.224525    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58653d9-6bc5-44c4-97ed-068232825e36-istio-token\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.231760 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.225341    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-istio-data\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.231760 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.227355    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58653d9-6bc5-44c4-97ed-068232825e36-istiod-ca-cert\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.231760 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.227625    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-credential-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.231760 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.228187    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-workload-certs\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.231760 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.228452    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-workload-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.231760 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.231353    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-istio-envoy\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.232413 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.232390    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58653d9-6bc5-44c4-97ed-068232825e36-istio-podinfo\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.249640 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.249607    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58653d9-6bc5-44c4-97ed-068232825e36-istio-token\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.251258 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.251231    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xncl6\" (UniqueName: \"kubernetes.io/projected/f58653d9-6bc5-44c4-97ed-068232825e36-kube-api-access-xncl6\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.525310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.514255    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:03.621804 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.621765    4962 generic.go:356] "Generic (PLEG): container finished" podID="977f25cb-ac5d-4290-a9a1-59322cc86f3d" containerID="1fa8cbaced73613c14dc2976e215755a76950eeb60e721b1279ce533213e469d" exitCode=0
Oct 29 22:10:03.623869 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.621860    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb" event={"ID":"977f25cb-ac5d-4290-a9a1-59322cc86f3d","Type":"ContainerDied","Data":"1fa8cbaced73613c14dc2976e215755a76950eeb60e721b1279ce533213e469d"}
Oct 29 22:10:03.631196 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.630588    4962 generic.go:356] "Generic (PLEG): container finished" podID="cbd6251b-df2f-4739-ae5d-8adbd277e52c" containerID="d738693af60f9ea4796875bb8a7eb853ab414ab98c8fe2bb56c17a1735111a63" exitCode=0
Oct 29 22:10:03.632512 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.632483    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b" event={"ID":"cbd6251b-df2f-4739-ae5d-8adbd277e52c","Type":"ContainerDied","Data":"d738693af60f9ea4796875bb8a7eb853ab414ab98c8fe2bb56c17a1735111a63"}
Oct 29 22:10:03.632780 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.632546    4962 scope.go:122] "RemoveContainer" containerID="dfc0898b2a96c3f21ca0edfc46c187b4a86426ecfa4fd8e31e709aab8893bdea"
Oct 29 22:10:03.744872 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.744846    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"]
Oct 29 22:10:03.754836 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.754804    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:10:03.842440 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.840637    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-credential-socket\") pod \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") "
Oct 29 22:10:03.842440 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.840715    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-podinfo\") pod \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") "
Oct 29 22:10:03.842440 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.840768    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-workload-certs\") pod \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") "
Oct 29 22:10:03.842440 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.840872    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-token\" (UniqueName: \"kubernetes.io/projected/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-token\") pod \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") "
Oct 29 22:10:03.842440 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.840988    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-envoy\") pod \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") "
Oct 29 22:10:03.842440 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.841052    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-data\") pod \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") "
Oct 29 22:10:03.842440 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.841167    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbd6251b-df2f-4739-ae5d-8adbd277e52c-kube-api-access-m86fx\" (UniqueName: \"kubernetes.io/projected/cbd6251b-df2f-4739-ae5d-8adbd277e52c-kube-api-access-m86fx\") pod \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") "
Oct 29 22:10:03.842440 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.841356    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-workload-socket\") pod \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") "
Oct 29 22:10:03.842440 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.841396    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istiod-ca-cert\") pod \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\" (UID: \"cbd6251b-df2f-4739-ae5d-8adbd277e52c\") "
Oct 29 22:10:03.843492 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.843187    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istiod-ca-cert" pod "cbd6251b-df2f-4739-ae5d-8adbd277e52c" (UID: "cbd6251b-df2f-4739-ae5d-8adbd277e52c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:03.843922 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.843633    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-credential-socket" pod "cbd6251b-df2f-4739-ae5d-8adbd277e52c" (UID: "cbd6251b-df2f-4739-ae5d-8adbd277e52c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:03.845301 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.845142    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-workload-socket" pod "cbd6251b-df2f-4739-ae5d-8adbd277e52c" (UID: "cbd6251b-df2f-4739-ae5d-8adbd277e52c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:03.845301 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.845206    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-data" pod "cbd6251b-df2f-4739-ae5d-8adbd277e52c" (UID: "cbd6251b-df2f-4739-ae5d-8adbd277e52c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:03.845301 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.845255    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-workload-certs" pod "cbd6251b-df2f-4739-ae5d-8adbd277e52c" (UID: "cbd6251b-df2f-4739-ae5d-8adbd277e52c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:03.854729 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.854705    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-podinfo" pod "cbd6251b-df2f-4739-ae5d-8adbd277e52c" (UID: "cbd6251b-df2f-4739-ae5d-8adbd277e52c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:10:03.862719 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.862682    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd6251b-df2f-4739-ae5d-8adbd277e52c-kube-api-access-m86fx" pod "cbd6251b-df2f-4739-ae5d-8adbd277e52c" (UID: "cbd6251b-df2f-4739-ae5d-8adbd277e52c"). InnerVolumeSpecName "kube-api-access-m86fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:03.862860 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.862742    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-envoy" pod "cbd6251b-df2f-4739-ae5d-8adbd277e52c" (UID: "cbd6251b-df2f-4739-ae5d-8adbd277e52c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:03.862860 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.862760    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-token" pod "cbd6251b-df2f-4739-ae5d-8adbd277e52c" (UID: "cbd6251b-df2f-4739-ae5d-8adbd277e52c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:03.930062 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.930034    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:10:03.942787 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.942759    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-workload-socket\") pod \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") "
Oct 29 22:10:03.942946 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.942825    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-token\" (UniqueName: \"kubernetes.io/projected/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-token\") pod \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") "
Oct 29 22:10:03.942946 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.942860    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/977f25cb-ac5d-4290-a9a1-59322cc86f3d-kube-api-access-2wv4v\" (UniqueName: \"kubernetes.io/projected/977f25cb-ac5d-4290-a9a1-59322cc86f3d-kube-api-access-2wv4v\") pod \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") "
Oct 29 22:10:03.943085 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.943026    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-credential-socket\") pod \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") "
Oct 29 22:10:03.943144 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.943138    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-workload-certs\") pod \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") "
Oct 29 22:10:03.943216 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.943200    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-data\") pod \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") "
Oct 29 22:10:03.943451 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.943235    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-envoy\") pod \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") "
Oct 29 22:10:03.943622 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.943585    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-podinfo\") pod \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") "
Oct 29 22:10:03.943682 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.943659    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istiod-ca-cert\") pod \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\" (UID: \"977f25cb-ac5d-4290-a9a1-59322cc86f3d\") "
Oct 29 22:10:03.944978 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.944937    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.944978 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.944966    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.945174 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.944981    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.945174 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.944995    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.945174 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.945008    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.945174 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.945022    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.945174 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.945036    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.945174 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.945048    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbd6251b-df2f-4739-ae5d-8adbd277e52c-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.945174 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.945063    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m86fx\" (UniqueName: \"kubernetes.io/projected/cbd6251b-df2f-4739-ae5d-8adbd277e52c-kube-api-access-m86fx\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:03.946609 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.945039    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-workload-socket" pod "977f25cb-ac5d-4290-a9a1-59322cc86f3d" (UID: "977f25cb-ac5d-4290-a9a1-59322cc86f3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:03.946721 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.945236    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-data" pod "977f25cb-ac5d-4290-a9a1-59322cc86f3d" (UID: "977f25cb-ac5d-4290-a9a1-59322cc86f3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:03.946721 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.945537    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-credential-socket" pod "977f25cb-ac5d-4290-a9a1-59322cc86f3d" (UID: "977f25cb-ac5d-4290-a9a1-59322cc86f3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:03.946721 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.946023    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-workload-certs" pod "977f25cb-ac5d-4290-a9a1-59322cc86f3d" (UID: "977f25cb-ac5d-4290-a9a1-59322cc86f3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:03.946721 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.946245    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istiod-ca-cert" pod "977f25cb-ac5d-4290-a9a1-59322cc86f3d" (UID: "977f25cb-ac5d-4290-a9a1-59322cc86f3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:03.946950 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.946790    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-token" pod "977f25cb-ac5d-4290-a9a1-59322cc86f3d" (UID: "977f25cb-ac5d-4290-a9a1-59322cc86f3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:03.947652 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.947615    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/977f25cb-ac5d-4290-a9a1-59322cc86f3d-kube-api-access-2wv4v" pod "977f25cb-ac5d-4290-a9a1-59322cc86f3d" (UID: "977f25cb-ac5d-4290-a9a1-59322cc86f3d"). InnerVolumeSpecName "kube-api-access-2wv4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:03.947753 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.947674    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-envoy" pod "977f25cb-ac5d-4290-a9a1-59322cc86f3d" (UID: "977f25cb-ac5d-4290-a9a1-59322cc86f3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:03.971468 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:03.971436    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-podinfo" pod "977f25cb-ac5d-4290-a9a1-59322cc86f3d" (UID: "977f25cb-ac5d-4290-a9a1-59322cc86f3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:10:04.096387 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.095037    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:04.096387 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.095083    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:04.096387 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.095100    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wv4v\" (UniqueName: \"kubernetes.io/projected/977f25cb-ac5d-4290-a9a1-59322cc86f3d-kube-api-access-2wv4v\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:04.096387 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.095118    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:04.096387 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.095136    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:04.096387 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.095150    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:04.096387 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.095165    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:04.096387 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.095180    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:04.096387 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.095194    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/977f25cb-ac5d-4290-a9a1-59322cc86f3d-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:04.192310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.192226    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:10:04.192991 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.192864    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:10:04.194016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.193951    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:10:04.202099 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.201978    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"]
Oct 29 22:10:04.281693 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.261900    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:10:04.298369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.297617    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-istio-envoy\") pod \"b0573e45-68c1-4509-abae-91401555180e\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") "
Oct 29 22:10:04.298369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.297650    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0573e45-68c1-4509-abae-91401555180e-kube-api-access-m8p78\" (UniqueName: \"kubernetes.io/projected/b0573e45-68c1-4509-abae-91401555180e-kube-api-access-m8p78\") pod \"b0573e45-68c1-4509-abae-91401555180e\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") "
Oct 29 22:10:04.298369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.297681    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-workload-certs\") pod \"b0573e45-68c1-4509-abae-91401555180e\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") "
Oct 29 22:10:04.298369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.297716    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-workload-socket\") pod \"b0573e45-68c1-4509-abae-91401555180e\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") "
Oct 29 22:10:04.298369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.297737    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0573e45-68c1-4509-abae-91401555180e-istio-token\" (UniqueName: \"kubernetes.io/projected/b0573e45-68c1-4509-abae-91401555180e-istio-token\") pod \"b0573e45-68c1-4509-abae-91401555180e\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") "
Oct 29 22:10:04.298369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.297841    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0573e45-68c1-4509-abae-91401555180e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0573e45-68c1-4509-abae-91401555180e-istiod-ca-cert\") pod \"b0573e45-68c1-4509-abae-91401555180e\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") "
Oct 29 22:10:04.298369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.297883    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0573e45-68c1-4509-abae-91401555180e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0573e45-68c1-4509-abae-91401555180e-istio-podinfo\") pod \"b0573e45-68c1-4509-abae-91401555180e\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") "
Oct 29 22:10:04.298369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.298014    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-istio-data\") pod \"b0573e45-68c1-4509-abae-91401555180e\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") "
Oct 29 22:10:04.298369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.298088    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-credential-socket\") pod \"b0573e45-68c1-4509-abae-91401555180e\" (UID: \"b0573e45-68c1-4509-abae-91401555180e\") "
Oct 29 22:10:04.298369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.298183    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-workload-certs" pod "b0573e45-68c1-4509-abae-91401555180e" (UID: "b0573e45-68c1-4509-abae-91401555180e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:04.298756 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.298636    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-workload-socket" pod "b0573e45-68c1-4509-abae-91401555180e" (UID: "b0573e45-68c1-4509-abae-91401555180e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:04.298919 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.298828    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-credential-socket" pod "b0573e45-68c1-4509-abae-91401555180e" (UID: "b0573e45-68c1-4509-abae-91401555180e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:04.298919 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.298879    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0573e45-68c1-4509-abae-91401555180e-istiod-ca-cert" pod "b0573e45-68c1-4509-abae-91401555180e" (UID: "b0573e45-68c1-4509-abae-91401555180e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:04.299212 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.299183    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-istio-data" pod "b0573e45-68c1-4509-abae-91401555180e" (UID: "b0573e45-68c1-4509-abae-91401555180e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:04.300034 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.299976    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:04.300034 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.300000    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:04.300416 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.300017    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0573e45-68c1-4509-abae-91401555180e-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:04.300416 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.300381    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-istio-envoy" pod "b0573e45-68c1-4509-abae-91401555180e" (UID: "b0573e45-68c1-4509-abae-91401555180e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:04.300676 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.300611    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:04.300967 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.300765    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0573e45-68c1-4509-abae-91401555180e-kube-api-access-m8p78" pod "b0573e45-68c1-4509-abae-91401555180e" (UID: "b0573e45-68c1-4509-abae-91401555180e"). InnerVolumeSpecName "kube-api-access-m8p78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:04.301112 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.301086    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:04.301415 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.301382    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0573e45-68c1-4509-abae-91401555180e-istio-podinfo" pod "b0573e45-68c1-4509-abae-91401555180e" (UID: "b0573e45-68c1-4509-abae-91401555180e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:10:04.301851 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.301806    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0573e45-68c1-4509-abae-91401555180e-istio-token" pod "b0573e45-68c1-4509-abae-91401555180e" (UID: "b0573e45-68c1-4509-abae-91401555180e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:04.403442 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.403408    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0573e45-68c1-4509-abae-91401555180e-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:04.403442 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.403442    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0573e45-68c1-4509-abae-91401555180e-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:04.403633 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.403458    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8p78\" (UniqueName: \"kubernetes.io/projected/b0573e45-68c1-4509-abae-91401555180e-kube-api-access-m8p78\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:04.403633 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.403473    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0573e45-68c1-4509-abae-91401555180e-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:04.641421 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.641323    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" event={"ID":"f58653d9-6bc5-44c4-97ed-068232825e36","Type":"ContainerStarted","Data":"9f620f46dca7e8791d502112c19efe3cdadb244d25b8bb4a1f8e00fd516a3a2f"}
Oct 29 22:10:04.641421 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.641411    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" event={"ID":"f58653d9-6bc5-44c4-97ed-068232825e36","Type":"ContainerStarted","Data":"49e79333805473445908f9bd94cee5ecf7f8eef0416235090a7e3c1fa66b8b05"}
Oct 29 22:10:04.643159 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.643132    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866" event={"ID":"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb","Type":"ContainerStarted","Data":"6a820430aed12e8315bdc2f3c23dda981159dd1974cde4d43239e17c77083a1c"}
Oct 29 22:10:04.643332 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.643172    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866" event={"ID":"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb","Type":"ContainerStarted","Data":"61cc5e3f1e618b95fe50f40a854b67eebc88a246e145d07015b263d314367047"}
Oct 29 22:10:04.643332 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.643229    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"
Oct 29 22:10:04.644868 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.644845    4962 generic.go:356] "Generic (PLEG): container finished" podID="a8ab68a7-4c08-494f-95c7-006502508963" containerID="146a99d6af0771756872912bf2cb97a8bcf63c3025bec77e33ac492b522fd026" exitCode=0
Oct 29 22:10:04.644960 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.644895    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf" event={"ID":"a8ab68a7-4c08-494f-95c7-006502508963","Type":"ContainerDied","Data":"146a99d6af0771756872912bf2cb97a8bcf63c3025bec77e33ac492b522fd026"}
Oct 29 22:10:04.646463 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.646442    4962 generic.go:356] "Generic (PLEG): container finished" podID="b0573e45-68c1-4509-abae-91401555180e" containerID="48a68d1dee50f13e4f72a6d12b111253b589e6af1052ac6ace6e4e22d806a140" exitCode=0
Oct 29 22:10:04.646538 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.646479    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"
Oct 29 22:10:04.646538 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.646520    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d" event={"ID":"b0573e45-68c1-4509-abae-91401555180e","Type":"ContainerDied","Data":"48a68d1dee50f13e4f72a6d12b111253b589e6af1052ac6ace6e4e22d806a140"}
Oct 29 22:10:04.646668 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.646568    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d" event={"ID":"b0573e45-68c1-4509-abae-91401555180e","Type":"ContainerDied","Data":"f247d92bdde14569d143219838c8b079336a5268ce003e845b4caf476f7823c3"}
Oct 29 22:10:04.646668 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.646597    4962 scope.go:122] "RemoveContainer" containerID="48a68d1dee50f13e4f72a6d12b111253b589e6af1052ac6ace6e4e22d806a140"
Oct 29 22:10:04.648058 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.648039    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"
Oct 29 22:10:04.648058 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.648049    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb" event={"ID":"977f25cb-ac5d-4290-a9a1-59322cc86f3d","Type":"ContainerDied","Data":"4bf81f5f71b9bba76c70d6faeeeb98fd53d0438c0f2931811da4cc4f919b34ca"}
Oct 29 22:10:04.649819 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.649787    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b" event={"ID":"cbd6251b-df2f-4739-ae5d-8adbd277e52c","Type":"ContainerDied","Data":"ffcb58f56f475baa91bbaa964f4c54814b644a456b8f9e8141711d696aec6674"}
Oct 29 22:10:04.649930 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.649829    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"
Oct 29 22:10:04.661867 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.661789    4962 scope.go:122] "RemoveContainer" containerID="48a68d1dee50f13e4f72a6d12b111253b589e6af1052ac6ace6e4e22d806a140"
Oct 29 22:10:04.665572 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:10:04.663183    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48a68d1dee50f13e4f72a6d12b111253b589e6af1052ac6ace6e4e22d806a140\": container with ID starting with 48a68d1dee50f13e4f72a6d12b111253b589e6af1052ac6ace6e4e22d806a140 not found: ID does not exist" containerID="48a68d1dee50f13e4f72a6d12b111253b589e6af1052ac6ace6e4e22d806a140"
Oct 29 22:10:04.665572 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.663221    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48a68d1dee50f13e4f72a6d12b111253b589e6af1052ac6ace6e4e22d806a140"} err="failed to get container status \"48a68d1dee50f13e4f72a6d12b111253b589e6af1052ac6ace6e4e22d806a140\": rpc error: code = NotFound desc = could not find container \"48a68d1dee50f13e4f72a6d12b111253b589e6af1052ac6ace6e4e22d806a140\": container with ID starting with 48a68d1dee50f13e4f72a6d12b111253b589e6af1052ac6ace6e4e22d806a140 not found: ID does not exist"
Oct 29 22:10:04.665572 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.663264    4962 scope.go:122] "RemoveContainer" containerID="1fa8cbaced73613c14dc2976e215755a76950eeb60e721b1279ce533213e469d"
Oct 29 22:10:04.677909 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.677893    4962 scope.go:122] "RemoveContainer" containerID="d738693af60f9ea4796875bb8a7eb853ab414ab98c8fe2bb56c17a1735111a63"
Oct 29 22:10:04.691063 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.691022    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" podStartSLOduration=2.691009325 podStartE2EDuration="2.691009325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:04.686719343 +0000 UTC m=+1425.759676307" watchObservedRunningTime="2026-10-29 22:10:04.691009325 +0000 UTC m=+1425.763966283"
Oct 29 22:10:04.731733 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.731678    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866" podStartSLOduration=2.16282829 podStartE2EDuration="2.731664351s" totalImagesPullingTime="568.836061ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:04.730270412 +0000 UTC m=+1425.803227389" watchObservedRunningTime="2026-10-29 22:10:04.731664351 +0000 UTC m=+1425.804621374"
Oct 29 22:10:04.770177 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.770152    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"]
Oct 29 22:10:04.776298 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.776266    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-ppfkb"]
Oct 29 22:10:04.810299 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.809759    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"]
Oct 29 22:10:04.814899 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.814873    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-pc65d"]
Oct 29 22:10:04.936961 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:04.926053    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"]
Oct 29 22:10:05.094581 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.094538    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="977f25cb-ac5d-4290-a9a1-59322cc86f3d" path="/var/lib/kubelet/pods/977f25cb-ac5d-4290-a9a1-59322cc86f3d/volumes"
Oct 29 22:10:05.095702 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.095678    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0573e45-68c1-4509-abae-91401555180e" path="/var/lib/kubelet/pods/b0573e45-68c1-4509-abae-91401555180e/volumes"
Oct 29 22:10:05.096174 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.096152    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h924b"]
Oct 29 22:10:05.285683 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.285654    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:10:05.345578 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.345550    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8ab68a7-4c08-494f-95c7-006502508963-istio-token\" (UniqueName: \"kubernetes.io/projected/a8ab68a7-4c08-494f-95c7-006502508963-istio-token\") pod \"a8ab68a7-4c08-494f-95c7-006502508963\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") "
Oct 29 22:10:05.345736 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.345610    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-istio-data\") pod \"a8ab68a7-4c08-494f-95c7-006502508963\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") "
Oct 29 22:10:05.345736 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.345703    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-workload-socket\") pod \"a8ab68a7-4c08-494f-95c7-006502508963\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") "
Oct 29 22:10:05.345822 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.345786    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8ab68a7-4c08-494f-95c7-006502508963-kube-api-access-ch7zs\" (UniqueName: \"kubernetes.io/projected/a8ab68a7-4c08-494f-95c7-006502508963-kube-api-access-ch7zs\") pod \"a8ab68a7-4c08-494f-95c7-006502508963\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") "
Oct 29 22:10:05.345947 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.345930    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-workload-certs\") pod \"a8ab68a7-4c08-494f-95c7-006502508963\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") "
Oct 29 22:10:05.346036 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.345959    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-istio-data" pod "a8ab68a7-4c08-494f-95c7-006502508963" (UID: "a8ab68a7-4c08-494f-95c7-006502508963"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:05.346036 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.345995    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8ab68a7-4c08-494f-95c7-006502508963-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ab68a7-4c08-494f-95c7-006502508963-istiod-ca-cert\") pod \"a8ab68a7-4c08-494f-95c7-006502508963\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") "
Oct 29 22:10:05.346036 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.346022    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-istio-envoy\") pod \"a8ab68a7-4c08-494f-95c7-006502508963\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") "
Oct 29 22:10:05.346036 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.346013    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-workload-socket" pod "a8ab68a7-4c08-494f-95c7-006502508963" (UID: "a8ab68a7-4c08-494f-95c7-006502508963"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:05.346246 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.346145    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8ab68a7-4c08-494f-95c7-006502508963-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ab68a7-4c08-494f-95c7-006502508963-istio-podinfo\") pod \"a8ab68a7-4c08-494f-95c7-006502508963\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") "
Oct 29 22:10:05.346246 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.346233    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-credential-socket\") pod \"a8ab68a7-4c08-494f-95c7-006502508963\" (UID: \"a8ab68a7-4c08-494f-95c7-006502508963\") "
Oct 29 22:10:05.346424 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.346248    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-workload-certs" pod "a8ab68a7-4c08-494f-95c7-006502508963" (UID: "a8ab68a7-4c08-494f-95c7-006502508963"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:05.346424 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.346370    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8ab68a7-4c08-494f-95c7-006502508963-istiod-ca-cert" pod "a8ab68a7-4c08-494f-95c7-006502508963" (UID: "a8ab68a7-4c08-494f-95c7-006502508963"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:05.346714 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.346695    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-credential-socket" pod "a8ab68a7-4c08-494f-95c7-006502508963" (UID: "a8ab68a7-4c08-494f-95c7-006502508963"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:05.346983 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.346960    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8ab68a7-4c08-494f-95c7-006502508963-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.347029 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.346989    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.347029 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.347004    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.347029 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.347015    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.347117 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.347026    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.348447 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.348422    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ab68a7-4c08-494f-95c7-006502508963-istio-token" pod "a8ab68a7-4c08-494f-95c7-006502508963" (UID: "a8ab68a7-4c08-494f-95c7-006502508963"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:05.348532 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.348460    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ab68a7-4c08-494f-95c7-006502508963-kube-api-access-ch7zs" pod "a8ab68a7-4c08-494f-95c7-006502508963" (UID: "a8ab68a7-4c08-494f-95c7-006502508963"). InnerVolumeSpecName "kube-api-access-ch7zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:05.348787 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.348766    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-istio-envoy" pod "a8ab68a7-4c08-494f-95c7-006502508963" (UID: "a8ab68a7-4c08-494f-95c7-006502508963"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:05.348836 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.348786    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8ab68a7-4c08-494f-95c7-006502508963-istio-podinfo" pod "a8ab68a7-4c08-494f-95c7-006502508963" (UID: "a8ab68a7-4c08-494f-95c7-006502508963"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:10:05.449398 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.449363    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8ab68a7-4c08-494f-95c7-006502508963-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.449398 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.449397    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8ab68a7-4c08-494f-95c7-006502508963-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.449645 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.449411    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8ab68a7-4c08-494f-95c7-006502508963-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.449645 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.449428    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch7zs\" (UniqueName: \"kubernetes.io/projected/a8ab68a7-4c08-494f-95c7-006502508963-kube-api-access-ch7zs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:05.519165 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.519077    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:05.520919 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.520893    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:10:05.521054 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.520946    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:10:05.671305 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.671260    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"
Oct 29 22:10:05.671772 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.671316    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf" event={"ID":"a8ab68a7-4c08-494f-95c7-006502508963","Type":"ContainerDied","Data":"23ebaabbc582d63d4a71ac1fe040ded52a3aceb73e783fd6c3d73bbbd67d5d5a"}
Oct 29 22:10:05.671772 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.671384    4962 scope.go:122] "RemoveContainer" containerID="146a99d6af0771756872912bf2cb97a8bcf63c3025bec77e33ac492b522fd026"
Oct 29 22:10:05.747237 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.747207    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"]
Oct 29 22:10:05.752066 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:05.752042    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-dd38c2-openshift-default-c8698fc87-h72vf"]
Oct 29 22:10:06.526021 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:06.525511    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:10:06.526021 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:06.525572    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:10:07.063676 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:07.063639    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8ab68a7-4c08-494f-95c7-006502508963" path="/var/lib/kubelet/pods/a8ab68a7-4c08-494f-95c7-006502508963/volumes"
Oct 29 22:10:07.064369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:07.064353    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbd6251b-df2f-4739-ae5d-8adbd277e52c" path="/var/lib/kubelet/pods/cbd6251b-df2f-4739-ae5d-8adbd277e52c/volumes"
Oct 29 22:10:07.541390 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:07.539737    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:10:07.541390 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:07.539804    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:10:08.689327 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:08.527306    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:10:08.689327 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:08.571380    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:10:08.752545 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:08.752509    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8"
Oct 29 22:10:09.311314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:09.309439    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:09.511432 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:09.511381    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:09.528955 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:09.528914    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:09.557645 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:09.553968    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:09.594362 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:09.588219    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:09.623320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:09.596225    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:10:09.625022 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:09.624979    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:10:09.786404 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:09.785630    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:09.856608 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:10:09.856518    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:10:09.886456 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:09.886415    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:10.372434 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:10.372382    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:10.431438 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:10.431387    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:10.521310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:10.518475    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:10.546313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:10.526438    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:10:10.546313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:10.528417    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:10:10.546313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:10.528470    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:10:10.594311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:10.549234    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:10.594311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:10.549625    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:10:10.594311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:10.549949    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 22:10:10.683336 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:10.669711    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:10.683336 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:10.670305    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:10:10.796476 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:10.795658    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:10:10.796476 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:10.796253    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 22:10:11.000419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:10.959408    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:10:11.000419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:10.959508    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": context deadline exceeded"
Oct 29 22:10:11.000419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:10.961038    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:10:11.003582 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:11.003538    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:10:11.276459 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:11.274187    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:11.508312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:11.483975    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:11.508312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:11.484610    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:10:11.582509 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:11.582400    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:10:11.645605 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:11.645557    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:10:11.670336 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:11.665714    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:10:11.858668 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:11.858564    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:11.879021 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:11.878976    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:10:11.916314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:11.913727    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"
Oct 29 22:10:11.916314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:11.913838    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:10:11.946311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:11.928072    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:10:12.008313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:11.983567    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:10:12.030439 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.009305    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 22:10:12.031957 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.028398    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded"
Oct 29 22:10:12.217940 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.137955    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:10:12.658470 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.360871    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"]
Oct 29 22:10:12.703329 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.702362    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:12.703329 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.702445    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw" podUID="f6631eff-5d71-483c-aac7-8d578274ce76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:12.703329 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.511313    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:10:12.703329 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.511221    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:10:12.715099 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.679708    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded"
Oct 29 22:10:12.715099 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.679873    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:10:12.762423 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.432280    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:10:12.775792 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.775755    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:10:12.807466 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.807423    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:10:12.807770 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.776026    4962 cpu_manager.go:414] "containerMap: removing container" podUID="cbd6251b-df2f-4739-ae5d-8adbd277e52c" containerName="istio-proxy"
Oct 29 22:10:12.807899 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.807884    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd6251b-df2f-4739-ae5d-8adbd277e52c" containerName="istio-proxy"
Oct 29 22:10:12.815591 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.815560    4962 cpu_manager.go:414] "containerMap: removing container" podUID="cbd6251b-df2f-4739-ae5d-8adbd277e52c" containerName="istio-proxy"
Oct 29 22:10:12.857035 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.856345    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd6251b-df2f-4739-ae5d-8adbd277e52c" containerName="istio-proxy"
Oct 29 22:10:12.861001 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.860974    4962 cpu_manager.go:414] "containerMap: removing container" podUID="b0573e45-68c1-4509-abae-91401555180e" containerName="istio-proxy"
Oct 29 22:10:12.861509 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.861489    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0573e45-68c1-4509-abae-91401555180e" containerName="istio-proxy"
Oct 29 22:10:12.861685 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.861670    4962 cpu_manager.go:414] "containerMap: removing container" podUID="a8ab68a7-4c08-494f-95c7-006502508963" containerName="istio-proxy"
Oct 29 22:10:12.861775 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.861765    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8ab68a7-4c08-494f-95c7-006502508963" containerName="istio-proxy"
Oct 29 22:10:12.861871 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.861862    4962 cpu_manager.go:414] "containerMap: removing container" podUID="977f25cb-ac5d-4290-a9a1-59322cc86f3d" containerName="istio-proxy"
Oct 29 22:10:12.861938 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.861930    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="977f25cb-ac5d-4290-a9a1-59322cc86f3d" containerName="istio-proxy"
Oct 29 22:10:12.862559 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.862513    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:12.919015 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.862634    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd6251b-df2f-4739-ae5d-8adbd277e52c" containerName="istio-proxy"
Oct 29 22:10:12.947740 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.917466    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:12.975417 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.964887    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0573e45-68c1-4509-abae-91401555180e" containerName="istio-proxy"
Oct 29 22:10:13.170651 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:13.170566    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8ab68a7-4c08-494f-95c7-006502508963" containerName="istio-proxy"
Oct 29 22:10:13.170871 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:13.170852    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="977f25cb-ac5d-4290-a9a1-59322cc86f3d" containerName="istio-proxy"
Oct 29 22:10:13.200655 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:13.044198    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:10:13.293993 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:12.974520    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:13.366331 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:13.044145    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:10:13.581543 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:13.577884    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:10:13.625310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:13.622522    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:10:13.913447 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:13.766195    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:10:13.913447 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:13.804675    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:10:13.913447 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:13.830059    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:10:13.913447 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:13.835462    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:10:13.939313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:13.935119    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:13.975095 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:13.975038    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:10:13.984334 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:13.977745    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:14.148953 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.148918    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 22:10:14.149982 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.149909    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:10:14.204364 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.189867    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f363a3\"/\"openshift-service-ca.crt\""
Oct 29 22:10:14.204364 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.190074    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f363a3\"/\"kube-root-ca.crt\""
Oct 29 22:10:14.212316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.212271    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f363a3\"/\"ipp-route-f363a3-kserve-self-signed-certs\""
Oct 29 22:10:14.263325 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.213264    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f363a3\"/\"default-dockercfg-rmxdn\""
Oct 29 22:10:14.306318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.283448    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-dshm\") pod \"ipp-route-f363a3-kserve-5bfbf4ccc-f8s67\" (UID: \"5ac13470-5074-4ded-8f00-e98621700e54\") " pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:10:14.307082 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.307048    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ac13470-5074-4ded-8f00-e98621700e54-tls-certs\") pod \"ipp-route-f363a3-kserve-5bfbf4ccc-f8s67\" (UID: \"5ac13470-5074-4ded-8f00-e98621700e54\") " pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:10:14.386314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.350480    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l267\" (UniqueName: \"kubernetes.io/projected/5ac13470-5074-4ded-8f00-e98621700e54-kube-api-access-9l267\") pod \"ipp-route-f363a3-kserve-5bfbf4ccc-f8s67\" (UID: \"5ac13470-5074-4ded-8f00-e98621700e54\") " pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:10:14.386314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.350630    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-home\") pod \"ipp-route-f363a3-kserve-5bfbf4ccc-f8s67\" (UID: \"5ac13470-5074-4ded-8f00-e98621700e54\") " pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:10:14.386314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.350694    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-tmp-dir\") pod \"ipp-route-f363a3-kserve-5bfbf4ccc-f8s67\" (UID: \"5ac13470-5074-4ded-8f00-e98621700e54\") " pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:10:14.386314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.350807    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-model-cache\") pod \"ipp-route-f363a3-kserve-5bfbf4ccc-f8s67\" (UID: \"5ac13470-5074-4ded-8f00-e98621700e54\") " pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:10:14.386314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.297489    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:14.386314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.297544    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:14.386314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.320896    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:14.413361 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.404158    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:14.673637 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.596504    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-dshm\") pod \"ipp-route-f363a3-kserve-5bfbf4ccc-f8s67\" (UID: \"5ac13470-5074-4ded-8f00-e98621700e54\") " pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:10:14.673637 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.596652    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ac13470-5074-4ded-8f00-e98621700e54-tls-certs\") pod \"ipp-route-f363a3-kserve-5bfbf4ccc-f8s67\" (UID: \"5ac13470-5074-4ded-8f00-e98621700e54\") " pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:10:14.673637 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.596805    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l267\" (UniqueName: \"kubernetes.io/projected/5ac13470-5074-4ded-8f00-e98621700e54-kube-api-access-9l267\") pod \"ipp-route-f363a3-kserve-5bfbf4ccc-f8s67\" (UID: \"5ac13470-5074-4ded-8f00-e98621700e54\") " pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:10:14.673637 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.596955    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-home\") pod \"ipp-route-f363a3-kserve-5bfbf4ccc-f8s67\" (UID: \"5ac13470-5074-4ded-8f00-e98621700e54\") " pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:10:14.673637 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.597016    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-tmp-dir\") pod \"ipp-route-f363a3-kserve-5bfbf4ccc-f8s67\" (UID: \"5ac13470-5074-4ded-8f00-e98621700e54\") " pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:10:14.673637 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.597135    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-model-cache\") pod \"ipp-route-f363a3-kserve-5bfbf4ccc-f8s67\" (UID: \"5ac13470-5074-4ded-8f00-e98621700e54\") " pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:10:14.673637 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.597947    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-model-cache\") pod \"ipp-route-f363a3-kserve-5bfbf4ccc-f8s67\" (UID: \"5ac13470-5074-4ded-8f00-e98621700e54\") " pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:10:14.673637 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.618532    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:10:14.673637 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.661215    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:10:14.767319 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.767142    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-dshm\") pod \"ipp-route-f363a3-kserve-5bfbf4ccc-f8s67\" (UID: \"5ac13470-5074-4ded-8f00-e98621700e54\") " pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:10:14.792313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.791080    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-tmp-dir\") pod \"ipp-route-f363a3-kserve-5bfbf4ccc-f8s67\" (UID: \"5ac13470-5074-4ded-8f00-e98621700e54\") " pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:10:14.800312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.797700    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded"
Oct 29 22:10:14.835906 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.835843    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:14.836979 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.836954    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-home\") pod \"ipp-route-f363a3-kserve-5bfbf4ccc-f8s67\" (UID: \"5ac13470-5074-4ded-8f00-e98621700e54\") " pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:10:14.925670 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.822439    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:14.988705 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:14.968815    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:10:15.030828 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:15.030788    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:15.140931 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:15.106597    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:10:15.140931 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:15.139470    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:10:15.175052 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:15.141267    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:10:15.206126 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:10:15.191895    4962 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.043s"
Oct 29 22:10:15.214546 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:15.214515    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"]
Oct 29 22:10:15.416516 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:15.416472    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:15.464362 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:15.416996    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:15.480313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:15.472610    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ac13470-5074-4ded-8f00-e98621700e54-tls-certs\") pod \"ipp-route-f363a3-kserve-5bfbf4ccc-f8s67\" (UID: \"5ac13470-5074-4ded-8f00-e98621700e54\") " pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:10:15.502384 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:15.417042    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:10:15.569315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:15.564202    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l267\" (UniqueName: \"kubernetes.io/projected/5ac13470-5074-4ded-8f00-e98621700e54-kube-api-access-9l267\") pod \"ipp-route-f363a3-kserve-5bfbf4ccc-f8s67\" (UID: \"5ac13470-5074-4ded-8f00-e98621700e54\") " pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:10:15.603100 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:15.603062    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:10:15.603676 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:15.603646    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:10:15.607397 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:15.607361    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:10:15.996312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:15.945425    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:10:16.096311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:16.021523    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:10:16.096311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:16.025937    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded"
Oct 29 22:10:16.096311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:16.026090    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:16.096311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:16.049766    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:10:16.108451 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:16.108381    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:16.148083 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:16.148029    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:16.148392 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:16.148360    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:10:16.464363 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:16.464318    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:16.480583 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:16.480530    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:16.484732 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:16.483363    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:16.519815 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:10:16.519782    4962 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.305s"
Oct 29 22:10:16.520110 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:16.520095    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 22:10:16.528477 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:16.528439    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:10:16.528723 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:16.528707    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"]
Oct 29 22:10:16.533562 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:16.533538    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"]
Oct 29 22:10:16.551797 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:16.551717    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt" podUID="4c60f25d-1a16-4cbb-bd55-42cf7e13946e" containerName="maas-api" containerID="cri-o://bcff9f095a02bf50417f7d2383986de72af5ce2bee19d478fc5df3df8ef9a124" gracePeriod=30
Oct 29 22:10:16.575448 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:16.556906    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:10:16.575772 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:16.575724    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:10:17.036326 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:17.029500    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:10:17.414254 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:17.044387    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:10:17.888388 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:17.888331    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:10:17.889049 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:17.124195    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:17.912322 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:17.165369    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:10:17.912322 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:17.166018    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:17.912322 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:17.166169    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:10:17.912322 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:17.044829    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:17.912322 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:17.899382    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:10:17.943929 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:17.943883    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:10:17.947441 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:17.947369    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:10:18.102678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:18.102612    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:18.128459 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:18.128400    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z" podUID="132ed775-c9e6-41f1-b224-fff550c6e073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:18.628546 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:18.128794    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:18.628546 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:18.628346    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2" podUID="03b2ca39-a1c7-45bf-8458-f23215d334b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:18.628546 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:18.390385    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:18.761877 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:18.151418    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:10:18.838315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:18.515571    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded"
Oct 29 22:10:18.913447 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:18.913398    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:18.914220 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:18.914188    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:19.063367 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:10:19.038152    4962 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.937s"
Oct 29 22:10:19.279494 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:19.172303    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:10:19.279494 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:19.177966    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:10:19.279494 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:18.545873    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:10:19.279494 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:19.200772    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:10:19.279494 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:18.390455    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:19.313313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:19.303945    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:10:19.313313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:19.304767    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:10:19.318886 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:19.318798    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:19.802959 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:19.802915    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:10:20.676314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:19.807873    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": context deadline exceeded"
Oct 29 22:10:20.676314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:19.880070    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:20.676314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:20.658671    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz" podUID="8054bc19-6db7-462f-8af0-335903ce62f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:20.676314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:19.933086    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:20.676314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:19.938552    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:10:20.676314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:19.981608    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:20.676314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:20.659879    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt" podUID="730edf47-8b07-477a-acfc-47faaaacfc84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:20.676314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:20.292827    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:10:20.676314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:20.056847    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:10:20.676314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:20.161819    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg" podUID="70e1bafe-4b93-44d7-a179-44d5a82fd069" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": context deadline exceeded"
Oct 29 22:10:20.676314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:20.292661    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:10:20.676314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:20.344979    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:10:20.676314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:20.660688    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:10:20.676314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:20.344889    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:10:20.676314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:20.359048    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt" podUID="4c60f25d-1a16-4cbb-bd55-42cf7e13946e" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.138:8443/health\": dial tcp 10.132.0.138:8443: connect: connection refused"
Oct 29 22:10:20.676314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:20.399398    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:10:20.708507 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:20.399554    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:10:20.708507 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:20.693787    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:20.721312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:20.712871    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:20.767313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:20.359520    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:20.781318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:20.585187    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:20.781318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:20.767617    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf" podUID="75cf3e4f-0488-4210-ab72-17b7afdeda6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:20.781318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:20.606429    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d" podUID="696bd863-08fa-4694-843f-ab65f3ccfeed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:20.795316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:20.790740    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:10:20.795316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:20.790801    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:10:20.872582 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:20.868653    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:20.872582 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:20.868797    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks" podUID="a85f7d35-4bff-45d0-a392-2ecf056c2051" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:21.234770 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:21.234345    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks" podUID="a85f7d35-4bff-45d0-a392-2ecf056c2051" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:21.302023 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:21.301938    4962 generic.go:356] "Generic (PLEG): container finished" podID="4c60f25d-1a16-4cbb-bd55-42cf7e13946e" containerID="bcff9f095a02bf50417f7d2383986de72af5ce2bee19d478fc5df3df8ef9a124" exitCode=0
Oct 29 22:10:21.433588 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:10:21.433550    4962 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.943s"
Oct 29 22:10:21.460448 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:21.460399    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:21.460718 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:21.460688    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5" podUID="a437f4e2-dc10-41de-851a-ee488b08c6f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:21.575360 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:21.550255    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:10:21.608328 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:21.603435    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:10:21.716535 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:21.664848    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:21.728314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:21.664895    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:10:21.733646 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:21.664918    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:10:21.741309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:21.664940    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:21.757497 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:21.664958    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:10:21.796186 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:21.664981    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:10:21.796963 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:21.664999    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:10:21.797350 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:21.665019    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:10:21.797876 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:21.778434    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:21.798341 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:21.798044    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:10:21.845369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:21.825652    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:22.197314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:22.173796    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt" event={"ID":"4c60f25d-1a16-4cbb-bd55-42cf7e13946e","Type":"ContainerDied","Data":"bcff9f095a02bf50417f7d2383986de72af5ce2bee19d478fc5df3df8ef9a124"}
Oct 29 22:10:22.197314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:22.173870    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:22.197314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:22.173905    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"]
Oct 29 22:10:22.371320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:22.366986    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd6251b-df2f-4739-ae5d-8adbd277e52c" containerName="istio-proxy"
Oct 29 22:10:22.456026 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:22.455989    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"]
Oct 29 22:10:22.549321 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:22.531689    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:10:22.708730 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:22.708624    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:10:22.708990 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:22.652959    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:22.709126 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:22.649559    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:22.781502 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:22.724646    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:23.027054 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:22.796154    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f363a3-openshift-default-dockercfg-6v2wv\""
Oct 29 22:10:23.091518 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:22.797206    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:23.112437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:22.797244    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:10:23.112437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:22.797264    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:10:23.112437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:22.797340    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:10:23.112437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:22.797364    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:10:23.112437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:22.797929    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:10:23.112437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:22.798166    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:23.112437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:22.801215    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:23.112437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:22.883523    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:23.112437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:22.916996    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.112437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.110046    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.112437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.110225    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.112437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.110462    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.112437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.110538    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfk4f\" (UniqueName: \"kubernetes.io/projected/9da507f9-8669-4b99-99de-8a7a83cd7850-kube-api-access-sfk4f\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.112437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.110619    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.112437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.110691    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.112437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.110756    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9da507f9-8669-4b99-99de-8a7a83cd7850-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.112437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.110807    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.112437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:22.918042    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"]
Oct 29 22:10:23.115195 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:22.918137    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:23.224564 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.224524    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-token\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:23.233657 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.233624    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-workload-certs\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:23.268312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.243403    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.700346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.697335    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfk4f\" (UniqueName: \"kubernetes.io/projected/9da507f9-8669-4b99-99de-8a7a83cd7850-kube-api-access-sfk4f\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.700346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.697410    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-envoy\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:23.700346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.697493    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.700346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.697541    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istiod-ca-cert\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:23.700346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.697596    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-data\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:23.700346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.697671    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.700346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.697754    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9da507f9-8669-4b99-99de-8a7a83cd7850-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.700346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.697809    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.700346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.697853    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-credential-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:23.700346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.697955    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.700346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.698018    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.734313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.254203    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.734313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.678335    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:10:23.734313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.709514    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:10:23.734313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.725732    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-podinfo\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:23.742315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.725924    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-workload-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:23.742315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.738446    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.742315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.738547    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fndv\" (UniqueName: \"kubernetes.io/projected/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-kube-api-access-6fndv\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:23.742315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.738778    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:23.742315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.739665    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.775424 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.748083    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.775424 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.771258    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.775424 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.773589    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"
Oct 29 22:10:23.801413 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.801379    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.856319 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.829731    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfk4f\" (UniqueName: \"kubernetes.io/projected/9da507f9-8669-4b99-99de-8a7a83cd7850-kube-api-access-sfk4f\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.896463 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.845873    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-openshift-service-ca\") pod \"4c60f25d-1a16-4cbb-bd55-42cf7e13946e\" (UID: \"4c60f25d-1a16-4cbb-bd55-42cf7e13946e\") "
Oct 29 22:10:23.896998 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.896963    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-maas-api-tls\") pod \"4c60f25d-1a16-4cbb-bd55-42cf7e13946e\" (UID: \"4c60f25d-1a16-4cbb-bd55-42cf7e13946e\") "
Oct 29 22:10:23.897659 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.859988    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9da507f9-8669-4b99-99de-8a7a83cd7850-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:23.897811 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.846261    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-openshift-service-ca" pod "4c60f25d-1a16-4cbb-bd55-42cf7e13946e" (UID: "4c60f25d-1a16-4cbb-bd55-42cf7e13946e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:23.900573 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.900551    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-kube-api-access-7bd99\" (UniqueName: \"kubernetes.io/projected/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-kube-api-access-7bd99\") pod \"4c60f25d-1a16-4cbb-bd55-42cf7e13946e\" (UID: \"4c60f25d-1a16-4cbb-bd55-42cf7e13946e\") "
Oct 29 22:10:23.928694 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.928667    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-metrics-certs\") pod \"4c60f25d-1a16-4cbb-bd55-42cf7e13946e\" (UID: \"4c60f25d-1a16-4cbb-bd55-42cf7e13946e\") "
Oct 29 22:10:23.964313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.929446    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-workload-certs\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:24.085313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.057939    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-envoy\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:24.085313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.058396    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istiod-ca-cert\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:24.085313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.058625    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-data\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:24.085313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.059127    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-credential-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:24.104318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.087978    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-podinfo\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:24.104318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.088264    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-workload-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:24.104318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.088498    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fndv\" (UniqueName: \"kubernetes.io/projected/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-kube-api-access-6fndv\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:24.104318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.088802    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-token\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:24.104318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.092499    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:24.108411 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.108369    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:24.116309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.112679    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:10:24.148319 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.143000    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:10:24.148839 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.143038    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:10:24.239832 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.143060    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:10:24.239832 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.143081    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:10:24.239832 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.143100    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:10:24.239832 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.143119    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:10:24.239832 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.143138    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:10:24.239832 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.934572    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-workload-certs\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:24.265342 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.265270    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded"
Oct 29 22:10:24.351794 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:23.941803    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:24.352784 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.352753    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istiod-ca-cert\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:24.353237 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.353214    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-data\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:24.354035 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.354013    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-credential-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:24.408367 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.407980    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-workload-socket\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:24.504313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.464417    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": context deadline exceeded"
Oct 29 22:10:24.504313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.499014    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fndv\" (UniqueName: \"kubernetes.io/projected/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-kube-api-access-6fndv\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:24.510862 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.510827    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-token\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:24.586756 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.586716    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-envoy\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:24.620340 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.610782    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-metrics-certs" pod "4c60f25d-1a16-4cbb-bd55-42cf7e13946e" (UID: "4c60f25d-1a16-4cbb-bd55-42cf7e13946e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:10:24.620340 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.611305    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:10:24.620340 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.611359    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:10:24.769552 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.749094    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:24.798356 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.770658    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-podinfo\") pod \"e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") " pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:24.798356 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.775031    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:24.885315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:24.859935    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:25.040932 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:25.040827    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:25.502392 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:25.098256    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-kube-api-access-7bd99" pod "4c60f25d-1a16-4cbb-bd55-42cf7e13946e" (UID: "4c60f25d-1a16-4cbb-bd55-42cf7e13946e"). InnerVolumeSpecName "kube-api-access-7bd99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:25.502392 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:25.152379    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:25.502392 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:25.170850    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-maas-api-tls" pod "4c60f25d-1a16-4cbb-bd55-42cf7e13946e" (UID: "4c60f25d-1a16-4cbb-bd55-42cf7e13946e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:10:25.502392 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:10:25.171956    4962 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.061s"
Oct 29 22:10:25.502392 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:25.190840    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:10:25.502392 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:25.196762    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:25.502392 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:25.197559    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:10:25.502392 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:25.198196    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:10:25.502392 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:25.309605    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:25.776689 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:25.776556    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:10:25.778303 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:25.778257    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:25.879562 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:25.879514    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:10:25.894343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:25.892215    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:25.894343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:25.892253    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bd99\" (UniqueName: \"kubernetes.io/projected/4c60f25d-1a16-4cbb-bd55-42cf7e13946e-kube-api-access-7bd99\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:26.043954 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:26.040706    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:10:26.153321 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:26.136627    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"
Oct 29 22:10:26.153321 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:26.147388    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:10:26.223380 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:26.211978    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:26.301448 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:26.249396    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:26.301448 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:26.250253    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:10:26.301448 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:26.280788    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:10:26.301448 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:26.281130    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:26.686310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:26.668469    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 29 22:10:26.686310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:26.668539    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 29 22:10:26.930303 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:26.930180    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:10:26.967577 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:26.944161    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:10:27.130434 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.130392    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:27.140315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.137886    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:10:27.184197 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.182407    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:10:27.252339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.252303    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:10:27.255142 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.255115    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:27.333314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.331753    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:27.333314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.332143    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:10:27.366698 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:10:27.366667    4962 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.195s"
Oct 29 22:10:27.367214 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.367196    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 22:10:27.384849 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.384813    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" event={"ID":"5ac13470-5074-4ded-8f00-e98621700e54","Type":"ContainerStarted","Data":"cb0a03ce7a2cf7b1ab85997ac46d016a55db67500339795ed95aba3299b145f8"}
Oct 29 22:10:27.391369 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:10:27.380786    4962 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c60f25d_1a16_4cbb_bd55_42cf7e13946e.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c60f25d_1a16_4cbb_bd55_42cf7e13946e.slice/crio-26342ad0be649d59c764a31abbec5360a796c92acc8e43b4099fa5f0e0eb3772\": RecentStats: unable to find data in memory cache]"
Oct 29 22:10:27.488944 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.488896    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"]
Oct 29 22:10:27.489226 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.489186    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt" event={"ID":"4c60f25d-1a16-4cbb-bd55-42cf7e13946e","Type":"ContainerDied","Data":"26342ad0be649d59c764a31abbec5360a796c92acc8e43b4099fa5f0e0eb3772"}
Oct 29 22:10:27.489718 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.489682    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 22:10:27.489908 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.489895    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:10:27.490040 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.490026    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"]
Oct 29 22:10:27.490237 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.490224    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 22:10:27.490374 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.490363    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"]
Oct 29 22:10:27.491146 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.491125    4962 scope.go:122] "RemoveContainer" containerID="bcff9f095a02bf50417f7d2383986de72af5ce2bee19d478fc5df3df8ef9a124"
Oct 29 22:10:27.500501 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.500478    4962 cpu_manager.go:414] "containerMap: removing container" podUID="4c60f25d-1a16-4cbb-bd55-42cf7e13946e" containerName="maas-api"
Oct 29 22:10:27.500637 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.500623    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c60f25d-1a16-4cbb-bd55-42cf7e13946e" containerName="maas-api"
Oct 29 22:10:27.500984 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.500965    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c60f25d-1a16-4cbb-bd55-42cf7e13946e" containerName="maas-api"
Oct 29 22:10:27.695102 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.693033    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:27.712491 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.699412    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:27.712491 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.699450    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 22:10:27.712491 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.699559    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"]
Oct 29 22:10:27.742789 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.737948    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:10:27.743336 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.743280    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"]
Oct 29 22:10:27.743519 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.743505    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"]
Oct 29 22:10:27.743705 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.743692    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"]
Oct 29 22:10:27.743811 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.743799    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-df42c7-5669f4bcb9-bgsgt"]
Oct 29 22:10:27.752339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.738904    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.752339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.745750    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 22:10:27.752339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.748465    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:27.756041 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.756001    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"]
Oct 29 22:10:27.762402 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.761414    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:27.762402 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.761536    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:27.762402 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.761566    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:27.762402 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.761592    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:27.762402 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.761646    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:27.762402 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.761712    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:27.762402 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.761783    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jc5c\" (UniqueName: \"kubernetes.io/projected/c1d2bd1d-783b-415a-8807-8f53396ec0ce-kube-api-access-4jc5c\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:27.762402 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.761860    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.762402 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.761895    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hvrp\" (UniqueName: \"kubernetes.io/projected/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-kube-api-access-6hvrp\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.762402 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.761960    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.762402 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.762005    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.762402 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.762138    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.762402 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.762241    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.762402 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.762273    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.762402 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.762400    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.763335 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.762446    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:27.771260 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.770884    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"]
Oct 29 22:10:27.771260 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.770998    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"
Oct 29 22:10:27.804650 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.803493    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.804650 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.803617    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:27.817807 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.817780    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:10:27.840328 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.840261    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-48c52b-serving-cert\""
Oct 29 22:10:27.856967 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.856944    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"]
Oct 29 22:10:27.857569 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.857539    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs" podUID="1693a089-535d-4f1d-815b-075639d9b9ab" containerName="istio-proxy" containerID="cri-o://0c2aff4fe1efcef4e14fea0b1742337c09c48cecb057ca7936e78554cb7434ad" gracePeriod=30
Oct 29 22:10:27.861955 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.861927    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"]
Oct 29 22:10:27.869193 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.869171    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"]
Oct 29 22:10:27.925445 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.924799    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:10:27.928311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.928268    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hvrp\" (UniqueName: \"kubernetes.io/projected/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-kube-api-access-6hvrp\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.928591 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.928576    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.928798 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.928777    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.929056 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.929030    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.929422 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.929207    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.929791 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.929769    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.929971 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.929956    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.931683 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.930842    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.933618 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.933553    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:27.933973 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.933956    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e577b68-1188-430f-9930-eb43bc82dd07-openshift-service-ca\") pod \"maas-api-e2e-api-b-48c52b-864449db78-t4tqp\" (UID: \"5e577b68-1188-430f-9930-eb43bc82dd07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"
Oct 29 22:10:27.934621 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.934597    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.934703 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.934671    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:27.934758 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.934736    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:27.934820 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.934805    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btcr6\" (UniqueName: \"kubernetes.io/projected/5e577b68-1188-430f-9930-eb43bc82dd07-kube-api-access-btcr6\") pod \"maas-api-e2e-api-b-48c52b-864449db78-t4tqp\" (UID: \"5e577b68-1188-430f-9930-eb43bc82dd07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"
Oct 29 22:10:27.934883 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.934857    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e577b68-1188-430f-9930-eb43bc82dd07-metrics-certs\") pod \"maas-api-e2e-api-b-48c52b-864449db78-t4tqp\" (UID: \"5e577b68-1188-430f-9930-eb43bc82dd07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"
Oct 29 22:10:27.934935 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.934898    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e577b68-1188-430f-9930-eb43bc82dd07-maas-api-tls\") pod \"maas-api-e2e-api-b-48c52b-864449db78-t4tqp\" (UID: \"5e577b68-1188-430f-9930-eb43bc82dd07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"
Oct 29 22:10:27.934935 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.934926    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:27.935035 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.934952    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:27.935035 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.934974    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:27.935035 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.935028    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:27.935192 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.935095    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:27.935192 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.935133    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jc5c\" (UniqueName: \"kubernetes.io/projected/c1d2bd1d-783b-415a-8807-8f53396ec0ce-kube-api-access-4jc5c\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:27.935319 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.935199    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.935614 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.935595    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.935845 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.935827    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.936409 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.936389    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.937345 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.937308    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.938049 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.938026    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-workload-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:27.938252 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.938235    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-workload-certs\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:27.938956 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.938935    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-credential-socket\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:27.939736 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.939712    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istiod-ca-cert\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:27.940355 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.940334    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-data\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:27.946732 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.946520    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.955144 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.955124    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:27.966185 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:27.966167    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:10:28.013836 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:28.013762    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-podinfo\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:28.015169 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:28.015138    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hvrp\" (UniqueName: \"kubernetes.io/projected/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-kube-api-access-6hvrp\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:28.016006 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:28.015974    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jc5c\" (UniqueName: \"kubernetes.io/projected/c1d2bd1d-783b-415a-8807-8f53396ec0ce-kube-api-access-4jc5c\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:28.024978 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:28.024921    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:28.038172 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:28.038149    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-token\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:28.038346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:28.038280    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-envoy\") pod \"e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") " pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:28.044796 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:28.043116    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btcr6\" (UniqueName: \"kubernetes.io/projected/5e577b68-1188-430f-9930-eb43bc82dd07-kube-api-access-btcr6\") pod \"maas-api-e2e-api-b-48c52b-864449db78-t4tqp\" (UID: \"5e577b68-1188-430f-9930-eb43bc82dd07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"
Oct 29 22:10:28.044796 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:28.043184    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e577b68-1188-430f-9930-eb43bc82dd07-metrics-certs\") pod \"maas-api-e2e-api-b-48c52b-864449db78-t4tqp\" (UID: \"5e577b68-1188-430f-9930-eb43bc82dd07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"
Oct 29 22:10:28.044796 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:28.043215    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e577b68-1188-430f-9930-eb43bc82dd07-maas-api-tls\") pod \"maas-api-e2e-api-b-48c52b-864449db78-t4tqp\" (UID: \"5e577b68-1188-430f-9930-eb43bc82dd07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"
Oct 29 22:10:28.044796 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:28.043644    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e577b68-1188-430f-9930-eb43bc82dd07-openshift-service-ca\") pod \"maas-api-e2e-api-b-48c52b-864449db78-t4tqp\" (UID: \"5e577b68-1188-430f-9930-eb43bc82dd07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"
Oct 29 22:10:28.044796 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:28.044670    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e577b68-1188-430f-9930-eb43bc82dd07-openshift-service-ca\") pod \"maas-api-e2e-api-b-48c52b-864449db78-t4tqp\" (UID: \"5e577b68-1188-430f-9930-eb43bc82dd07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"
Oct 29 22:10:28.045115 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:10:28.044986    4962 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-serving-cert: secret "maas-api-e2e-api-b-48c52b-serving-cert" not found
Oct 29 22:10:28.045115 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:10:28.045039    4962 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e577b68-1188-430f-9930-eb43bc82dd07-maas-api-tls podName:5e577b68-1188-430f-9930-eb43bc82dd07 nodeName:}" failed. No retries permitted until 2026-10-29 22:10:28.545020846 +0000 UTC m=+1449.617977795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5e577b68-1188-430f-9930-eb43bc82dd07-maas-api-tls") pod "maas-api-e2e-api-b-48c52b-864449db78-t4tqp" (UID: "5e577b68-1188-430f-9930-eb43bc82dd07") : secret "maas-api-e2e-api-b-48c52b-serving-cert" not found
Oct 29 22:10:28.049834 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:28.049085    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e577b68-1188-430f-9930-eb43bc82dd07-metrics-certs\") pod \"maas-api-e2e-api-b-48c52b-864449db78-t4tqp\" (UID: \"5e577b68-1188-430f-9930-eb43bc82dd07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"
Oct 29 22:10:28.354654 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:28.352000    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:28.956923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:28.956890    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:29.013311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:28.958795    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:10:29.013311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:29.009524    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:10:29.013311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:29.009631    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:10:29.013311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:28.959873    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:10:29.013311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:29.010343    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:10:29.013311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:29.010429    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:10:29.013311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:28.990038    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btcr6\" (UniqueName: \"kubernetes.io/projected/5e577b68-1188-430f-9930-eb43bc82dd07-kube-api-access-btcr6\") pod \"maas-api-e2e-api-b-48c52b-864449db78-t4tqp\" (UID: \"5e577b68-1188-430f-9930-eb43bc82dd07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"
Oct 29 22:10:29.013311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:28.973454    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"]
Oct 29 22:10:29.013311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:29.011107    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" event={"ID":"5ac13470-5074-4ded-8f00-e98621700e54","Type":"ContainerStarted","Data":"f09e58bbf1adfe9b64f3fd8d2e1b9fe2b8c5b9ca695f8e59103da252b7b18fee"}
Oct 29 22:10:29.013311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:29.011135    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"]
Oct 29 22:10:29.013311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:29.011174    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb" event={"ID":"692ec1aa-ccc0-4712-88bd-b555b8eb9dae","Type":"ContainerStarted","Data":"90679bae151a874ee40433f9ed4f83739df0e481cf6e8b0ed2d5602dfa4acfde"}
Oct 29 22:10:29.013311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:29.011211    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb" event={"ID":"9da507f9-8669-4b99-99de-8a7a83cd7850","Type":"ContainerStarted","Data":"3df7d5d64c5de0e691ceb32aececaf8b8ca56f4d806a50e228af730b99621238"}
Oct 29 22:10:29.013311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:28.977176    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e577b68-1188-430f-9930-eb43bc82dd07-maas-api-tls\") pod \"maas-api-e2e-api-b-48c52b-864449db78-t4tqp\" (UID: \"5e577b68-1188-430f-9930-eb43bc82dd07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"
Oct 29 22:10:29.017469 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:29.013927    4962 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" secret="" err="secret \"e2e-ipp-b-df42c7-openshift-default-dockercfg-jqslj\" not found"
Oct 29 22:10:29.021147 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:29.021123    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e577b68-1188-430f-9930-eb43bc82dd07-maas-api-tls\") pod \"maas-api-e2e-api-b-48c52b-864449db78-t4tqp\" (UID: \"5e577b68-1188-430f-9930-eb43bc82dd07\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"
Oct 29 22:10:29.129422 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:29.129391    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c60f25d-1a16-4cbb-bd55-42cf7e13946e" path="/var/lib/kubelet/pods/4c60f25d-1a16-4cbb-bd55-42cf7e13946e/volumes"
Oct 29 22:10:29.147570 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:29.147116    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"
Oct 29 22:10:29.317486 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:29.317341    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"]
Oct 29 22:10:29.369520 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:29.369481    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:10:29.370011 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:29.369983    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:10:29.370180 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:29.370162    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:10:29.400362 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:29.396804    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podStartSLOduration=18.396786956 podStartE2EDuration="18.396786956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:29.387160722 +0000 UTC m=+1450.460117690" watchObservedRunningTime="2026-10-29 22:10:29.396786956 +0000 UTC m=+1450.469743921"
Oct 29 22:10:29.430402 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:29.430369    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:10:29.430593 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:29.430574    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:10:29.430733 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:29.430718    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:10:29.432771 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:29.432739    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"]
Oct 29 22:10:29.670782 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:29.670479    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:10:29.884382 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:10:29.873917    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e577b68_1188_430f_9930_eb43bc82dd07.slice/crio-75de97a8d95537ba59eec77968856a9f5abb292e55b648641a387a96d22831dd WatchSource:0}: Error finding container 75de97a8d95537ba59eec77968856a9f5abb292e55b648641a387a96d22831dd: Status 404 returned error can't find the container with id 75de97a8d95537ba59eec77968856a9f5abb292e55b648641a387a96d22831dd
Oct 29 22:10:29.884382 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:29.875130    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"]
Oct 29 22:10:30.002353 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:29.999505    4962 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:10:30.038486 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:30.038448    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb" event={"ID":"692ec1aa-ccc0-4712-88bd-b555b8eb9dae","Type":"ContainerStarted","Data":"acdac0b3a761ff0b0b767da39f44cf8b0a77a0c3d758b7d5e4f8735def15f7dd"}
Oct 29 22:10:30.038681 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:30.038654    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb" podUID="692ec1aa-ccc0-4712-88bd-b555b8eb9dae" containerName="istio-proxy" containerID="cri-o://acdac0b3a761ff0b0b767da39f44cf8b0a77a0c3d758b7d5e4f8735def15f7dd" gracePeriod=30
Oct 29 22:10:30.078806 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:30.078151    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z" event={"ID":"4ad0fa83-03aa-4824-ae79-c47cf352ff6a","Type":"ContainerStarted","Data":"30ffe75ec432d510ca3d1cc3232e37e3966b0a89f17ad35abb6cd42c1a605612"}
Oct 29 22:10:30.078806 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:30.078215    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z" event={"ID":"4ad0fa83-03aa-4824-ae79-c47cf352ff6a","Type":"ContainerStarted","Data":"35b73dd9c03d27eb02282cf8ac82e7cd731df144fffbe8bd1b755898ee74b55a"}
Oct 29 22:10:30.083733 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:30.083697    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h" event={"ID":"c1d2bd1d-783b-415a-8807-8f53396ec0ce","Type":"ContainerStarted","Data":"7e91d64313600eb9dd1084f4c8b2dd124b3b18b960654cb634390d4a53f5a064"}
Oct 29 22:10:30.097401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:30.097366    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb" event={"ID":"9da507f9-8669-4b99-99de-8a7a83cd7850","Type":"ContainerStarted","Data":"818cb4a9e6555cfb76a34b6ac6fb6b2bd9e430b059d01e7e5a0659907bbdc053"}
Oct 29 22:10:30.097803 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:30.097769    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb" podStartSLOduration=12.097755777 podStartE2EDuration="12.097755777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:30.08581515 +0000 UTC m=+1451.158772126" watchObservedRunningTime="2026-10-29 22:10:30.097755777 +0000 UTC m=+1451.170712741"
Oct 29 22:10:30.104374 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:30.102457    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp" event={"ID":"5e577b68-1188-430f-9930-eb43bc82dd07","Type":"ContainerStarted","Data":"75de97a8d95537ba59eec77968856a9f5abb292e55b648641a387a96d22831dd"}
Oct 29 22:10:30.104374 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:30.102583    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy" containerID="cri-o://9f620f46dca7e8791d502112c19efe3cdadb244d25b8bb4a1f8e00fd516a3a2f" gracePeriod=30
Oct 29 22:10:30.140123 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:30.140058    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z" podStartSLOduration=11.140038735 podStartE2EDuration="11.140038735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:30.135775505 +0000 UTC m=+1451.208732477" watchObservedRunningTime="2026-10-29 22:10:30.140038735 +0000 UTC m=+1451.212995704"
Oct 29 22:10:30.196958 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:30.196897    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h" podStartSLOduration=11.196868316 podStartE2EDuration="11.196868316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:30.189983533 +0000 UTC m=+1451.262940497" watchObservedRunningTime="2026-10-29 22:10:30.196868316 +0000 UTC m=+1451.269825283"
Oct 29 22:10:30.237469 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:30.237408    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb" podStartSLOduration=12.237390443 podStartE2EDuration="12.237390443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:30.234739672 +0000 UTC m=+1451.307696636" watchObservedRunningTime="2026-10-29 22:10:30.237390443 +0000 UTC m=+1451.310347409"
Oct 29 22:10:30.417307 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:30.417001    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:30.424669 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:30.424469    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 29 22:10:30.424669 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:30.424536    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z" podUID="4ad0fa83-03aa-4824-ae79-c47cf352ff6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 29 22:10:30.958842 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:30.958783    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:31.070903 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:31.070845    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:31.102324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:31.081402    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 29 22:10:31.102324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:31.081465    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb" podUID="9da507f9-8669-4b99-99de-8a7a83cd7850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 29 22:10:31.232472 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:31.228810    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h" event={"ID":"c1d2bd1d-783b-415a-8807-8f53396ec0ce","Type":"ContainerStarted","Data":"88d8fe31202439750dfdaba647701549355ed02ad6eda9e0ed01494a4fe6e3fd"}
Oct 29 22:10:31.254132 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:31.242650    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp" event={"ID":"5e577b68-1188-430f-9930-eb43bc82dd07","Type":"ContainerStarted","Data":"0cecb9a41275548a1b3e011afdeeab9091e9e79bd6f49ba10c3aae8ea96f0e1f"}
Oct 29 22:10:31.254132 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:31.252282    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"
Oct 29 22:10:31.267697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:31.267652    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:10:31.267958 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:31.267710    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs" podUID="1693a089-535d-4f1d-815b-075639d9b9ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:10:31.361339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:31.360126    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp" podStartSLOduration=4.029151624 podStartE2EDuration="4.36010677s" totalImagesPullingTime="330.955146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:10:31.346866296 +0000 UTC m=+1452.419823261" watchObservedRunningTime="2026-10-29 22:10:31.36010677 +0000 UTC m=+1452.433063751"
Oct 29 22:10:31.415341 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:31.414817    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 29 22:10:31.415341 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:31.414900    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z" podUID="4ad0fa83-03aa-4824-ae79-c47cf352ff6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 29 22:10:32.135312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:32.126558    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 29 22:10:32.135312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:32.126618    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb" podUID="9da507f9-8669-4b99-99de-8a7a83cd7850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 29 22:10:32.160311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:32.157744    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 29 22:10:32.160311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:32.157802    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h" podUID="c1d2bd1d-783b-415a-8807-8f53396ec0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 29 22:10:32.463310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:32.460329    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 29 22:10:32.463310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:32.460399    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z" podUID="4ad0fa83-03aa-4824-ae79-c47cf352ff6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 29 22:10:32.502314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:32.502252    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:10:32.502533 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:32.502342    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn" podUID="bced75a3-37ba-4595-9dc0-b354574eba07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:32.960373 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:32.960340    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 29 22:10:32.960622 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:32.960592    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h" podUID="c1d2bd1d-783b-415a-8807-8f53396ec0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 29 22:10:33.146601 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:33.071352    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 29 22:10:33.146601 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:33.071412    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb" podUID="9da507f9-8669-4b99-99de-8a7a83cd7850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 29 22:10:33.417365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:33.413387    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 29 22:10:33.417365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:33.413441    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z" podUID="4ad0fa83-03aa-4824-ae79-c47cf352ff6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 29 22:10:33.521470 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:33.520768    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"]
Oct 29 22:10:33.812393 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:33.798685    4962 generic.go:356] "Generic (PLEG): container finished" podID="1693a089-535d-4f1d-815b-075639d9b9ab" containerID="0c2aff4fe1efcef4e14fea0b1742337c09c48cecb057ca7936e78554cb7434ad" exitCode=0
Oct 29 22:10:34.214309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:33.904011    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs" event={"ID":"1693a089-535d-4f1d-815b-075639d9b9ab","Type":"ContainerDied","Data":"0c2aff4fe1efcef4e14fea0b1742337c09c48cecb057ca7936e78554cb7434ad"}
Oct 29 22:10:34.240538 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:34.144094    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 29 22:10:34.240814 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:34.240776    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb" podUID="9da507f9-8669-4b99-99de-8a7a83cd7850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 29 22:10:34.280311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:34.144142    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 29 22:10:34.280744 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:34.280702    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h" podUID="c1d2bd1d-783b-415a-8807-8f53396ec0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 29 22:10:34.361315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:34.341455    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:34.525735 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:34.508455    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:34.526312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:34.468832    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:34.562401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:34.562368    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 29 22:10:34.562655 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:34.562630    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z" podUID="4ad0fa83-03aa-4824-ae79-c47cf352ff6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 29 22:10:34.657355 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:34.657310    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:10:34.723973 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:34.703460    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:34.843313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:34.841449    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:10:35.022149 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.022058    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:10:35.055341 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.022500    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 29 22:10:35.055341 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.024132    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h" podUID="c1d2bd1d-783b-415a-8807-8f53396ec0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 29 22:10:35.055341 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.022597    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:35.055341 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.022549    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:35.084313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.056546    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-istio-envoy\") pod \"1693a089-535d-4f1d-815b-075639d9b9ab\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") "
Oct 29 22:10:35.084313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.056619    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1693a089-535d-4f1d-815b-075639d9b9ab-istio-token\" (UniqueName: \"kubernetes.io/projected/1693a089-535d-4f1d-815b-075639d9b9ab-istio-token\") pod \"1693a089-535d-4f1d-815b-075639d9b9ab\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") "
Oct 29 22:10:35.084313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.056701    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1693a089-535d-4f1d-815b-075639d9b9ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1693a089-535d-4f1d-815b-075639d9b9ab-istiod-ca-cert\") pod \"1693a089-535d-4f1d-815b-075639d9b9ab\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") "
Oct 29 22:10:35.084313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.056770    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-credential-socket\") pod \"1693a089-535d-4f1d-815b-075639d9b9ab\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") "
Oct 29 22:10:35.084313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.056856    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1693a089-535d-4f1d-815b-075639d9b9ab-kube-api-access-tx67k\" (UniqueName: \"kubernetes.io/projected/1693a089-535d-4f1d-815b-075639d9b9ab-kube-api-access-tx67k\") pod \"1693a089-535d-4f1d-815b-075639d9b9ab\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") "
Oct 29 22:10:35.084313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.056911    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-workload-socket\") pod \"1693a089-535d-4f1d-815b-075639d9b9ab\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") "
Oct 29 22:10:35.084313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.057133    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-istio-data\") pod \"1693a089-535d-4f1d-815b-075639d9b9ab\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") "
Oct 29 22:10:35.084313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.057185    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1693a089-535d-4f1d-815b-075639d9b9ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1693a089-535d-4f1d-815b-075639d9b9ab-istio-podinfo\") pod \"1693a089-535d-4f1d-815b-075639d9b9ab\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") "
Oct 29 22:10:35.084313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.057261    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-workload-certs\") pod \"1693a089-535d-4f1d-815b-075639d9b9ab\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") "
Oct 29 22:10:35.096397 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.086682    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-workload-certs" pod "1693a089-535d-4f1d-815b-075639d9b9ab" (UID: "1693a089-535d-4f1d-815b-075639d9b9ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:35.096851 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.096638    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-istio-data" pod "1693a089-535d-4f1d-815b-075639d9b9ab" (UID: "1693a089-535d-4f1d-815b-075639d9b9ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:35.097360 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.097027    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-workload-socket" pod "1693a089-535d-4f1d-815b-075639d9b9ab" (UID: "1693a089-535d-4f1d-815b-075639d9b9ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:35.097850 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.097824    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-credential-socket" pod "1693a089-535d-4f1d-815b-075639d9b9ab" (UID: "1693a089-535d-4f1d-815b-075639d9b9ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:35.098056 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.098035    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 29 22:10:35.098221 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.098192    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb" podUID="9da507f9-8669-4b99-99de-8a7a83cd7850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 29 22:10:35.130323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.117946    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1693a089-535d-4f1d-815b-075639d9b9ab-istiod-ca-cert" pod "1693a089-535d-4f1d-815b-075639d9b9ab" (UID: "1693a089-535d-4f1d-815b-075639d9b9ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:35.211320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.179443    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:35.211320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.182735    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1693a089-535d-4f1d-815b-075639d9b9ab-istio-token" pod "1693a089-535d-4f1d-815b-075639d9b9ab" (UID: "1693a089-535d-4f1d-815b-075639d9b9ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:35.211320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.183361    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:35.211320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.201887    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1693a089-535d-4f1d-815b-075639d9b9ab-istio-token\" (UniqueName: \"kubernetes.io/projected/1693a089-535d-4f1d-815b-075639d9b9ab-istio-token\") pod \"1693a089-535d-4f1d-815b-075639d9b9ab\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") "
Oct 29 22:10:35.211320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.202846    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1693a089-535d-4f1d-815b-075639d9b9ab-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:35.211320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.202868    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:35.211320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.202884    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:35.211320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.202898    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:35.211320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.202911    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:35.211320 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:10:35.203005    4962 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/1693a089-535d-4f1d-815b-075639d9b9ab/volumes/kubernetes.io~projected/istio-token
Oct 29 22:10:35.211320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.203069    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1693a089-535d-4f1d-815b-075639d9b9ab-istio-token" pod "1693a089-535d-4f1d-815b-075639d9b9ab" (UID: "1693a089-535d-4f1d-815b-075639d9b9ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:35.311773 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.311677    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1693a089-535d-4f1d-815b-075639d9b9ab-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:35.488328 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.479611    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 29 22:10:35.488328 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.479670    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z" podUID="4ad0fa83-03aa-4824-ae79-c47cf352ff6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 29 22:10:35.511326 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.509552    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:35.585752 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.514490    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:35.585752 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.522450    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 22:10:35.585752 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.547723    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:35.585752 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.547897    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:10:35.628423 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.628391    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"
Oct 29 22:10:35.628758 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.628734    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs" event={"ID":"1693a089-535d-4f1d-815b-075639d9b9ab","Type":"ContainerDied","Data":"c66286813b5eaf4da8ac96a43aeaf9e4f7ca5215c485cd16d085e14f75c64621"}
Oct 29 22:10:35.628932 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.628917    4962 scope.go:122] "RemoveContainer" containerID="0c2aff4fe1efcef4e14fea0b1742337c09c48cecb057ca7936e78554cb7434ad"
Oct 29 22:10:35.665275 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.648511    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:10:35.671071 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.671044    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:10:35.772899 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.771472    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:35.772899 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.771792    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:35.772899 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.771938    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 22:10:35.772899 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:35.772003    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:10:36.038352 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.036450    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:36.038352 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.037915    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-istio-envoy" pod "1693a089-535d-4f1d-815b-075639d9b9ab" (UID: "1693a089-535d-4f1d-815b-075639d9b9ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:36.038352 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.038064    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1693a089-535d-4f1d-815b-075639d9b9ab-kube-api-access-tx67k" pod "1693a089-535d-4f1d-815b-075639d9b9ab" (UID: "1693a089-535d-4f1d-815b-075639d9b9ab"). InnerVolumeSpecName "kube-api-access-tx67k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:36.039108 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.038987    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1693a089-535d-4f1d-815b-075639d9b9ab-istio-podinfo" pod "1693a089-535d-4f1d-815b-075639d9b9ab" (UID: "1693a089-535d-4f1d-815b-075639d9b9ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:10:36.070231 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.039411    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:10:36.070231 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.040472    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 22:10:36.070231 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.040533    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 22:10:36.070231 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.041220    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Oct 29 22:10:36.070231 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.041503    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:10:36.119640 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.119605    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 29 22:10:36.147655 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.147593    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb" podUID="9da507f9-8669-4b99-99de-8a7a83cd7850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 29 22:10:36.214362 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.202975    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:10:36.242666 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.242247    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:10:36.288313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.249387    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1693a089-535d-4f1d-815b-075639d9b9ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1693a089-535d-4f1d-815b-075639d9b9ab-istio-podinfo\") pod \"1693a089-535d-4f1d-815b-075639d9b9ab\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") "
Oct 29 22:10:36.288313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.249552    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-istio-envoy\") pod \"1693a089-535d-4f1d-815b-075639d9b9ab\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") "
Oct 29 22:10:36.288313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.249696    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1693a089-535d-4f1d-815b-075639d9b9ab-kube-api-access-tx67k\" (UniqueName: \"kubernetes.io/projected/1693a089-535d-4f1d-815b-075639d9b9ab-kube-api-access-tx67k\") pod \"1693a089-535d-4f1d-815b-075639d9b9ab\" (UID: \"1693a089-535d-4f1d-815b-075639d9b9ab\") "
Oct 29 22:10:36.288313 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:10:36.250444    4962 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/1693a089-535d-4f1d-815b-075639d9b9ab/volumes/kubernetes.io~projected/kube-api-access-tx67k
Oct 29 22:10:36.288313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.250521    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1693a089-535d-4f1d-815b-075639d9b9ab-kube-api-access-tx67k" pod "1693a089-535d-4f1d-815b-075639d9b9ab" (UID: "1693a089-535d-4f1d-815b-075639d9b9ab"). InnerVolumeSpecName "kube-api-access-tx67k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:36.288313 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:10:36.250628    4962 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/1693a089-535d-4f1d-815b-075639d9b9ab/volumes/kubernetes.io~downward-api/istio-podinfo
Oct 29 22:10:36.288313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.250638    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1693a089-535d-4f1d-815b-075639d9b9ab-istio-podinfo" pod "1693a089-535d-4f1d-815b-075639d9b9ab" (UID: "1693a089-535d-4f1d-815b-075639d9b9ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:10:36.288313 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:10:36.250677    4962 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/1693a089-535d-4f1d-815b-075639d9b9ab/volumes/kubernetes.io~empty-dir/istio-envoy
Oct 29 22:10:36.288313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.250688    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-istio-envoy" pod "1693a089-535d-4f1d-815b-075639d9b9ab" (UID: "1693a089-535d-4f1d-815b-075639d9b9ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:36.336620 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.336574    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 29 22:10:36.337423 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.337278    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h" podUID="c1d2bd1d-783b-415a-8807-8f53396ec0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 29 22:10:36.428711 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.428656    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 29 22:10:36.525446 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.524890    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:36.525446 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.524995    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:10:36.597808 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.579481    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:36.597808 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.587755    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1693a089-535d-4f1d-815b-075639d9b9ab-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:36.597808 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.587795    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1693a089-535d-4f1d-815b-075639d9b9ab-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:36.597808 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.587811    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx67k\" (UniqueName: \"kubernetes.io/projected/1693a089-535d-4f1d-815b-075639d9b9ab-kube-api-access-tx67k\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:36.686941 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.686889    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z" podUID="4ad0fa83-03aa-4824-ae79-c47cf352ff6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 29 22:10:36.889716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.693105    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 22:10:36.889975 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.771803    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:36.897313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.771879    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:36.966311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.961349    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 29 22:10:36.966311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:36.961415    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h" podUID="c1d2bd1d-783b-415a-8807-8f53396ec0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 29 22:10:37.043462 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:37.043417    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:10:37.051319 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:37.044092    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:37.065935 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:37.051636    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:37.597329 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:37.597271    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 29 22:10:37.598449 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:37.598416    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb" podUID="9da507f9-8669-4b99-99de-8a7a83cd7850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 29 22:10:37.625321 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:37.608431    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:10:37.702198 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:37.702162    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:37.702706 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:37.702676    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:10:37.703031 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:37.703008    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:10:37.866504 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:37.866405    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 29 22:10:38.033539 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.033497    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z" podUID="4ad0fa83-03aa-4824-ae79-c47cf352ff6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 29 22:10:38.033801 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:37.921446    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:38.033922 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:37.963416    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 29 22:10:38.034071 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.034013    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h" podUID="c1d2bd1d-783b-415a-8807-8f53396ec0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 29 22:10:38.038214 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.038150    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:10:38.112229 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.040769    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Oct 29 22:10:38.112229 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.108620    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:10:38.172427 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.159150    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:38.198949 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.198915    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 29 22:10:38.199217 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.199189    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb" podUID="9da507f9-8669-4b99-99de-8a7a83cd7850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 29 22:10:38.261930 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.226392    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:10:38.261930 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.230567    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks" podUID="a85f7d35-4bff-45d0-a392-2ecf056c2051" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:38.444952 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.444860    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body=
Oct 29 22:10:38.445199 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.445167    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z" podUID="4ad0fa83-03aa-4824-ae79-c47cf352ff6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused"
Oct 29 22:10:38.678931 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.673343    4962 generic.go:356] "Generic (PLEG): container finished" podID="f58653d9-6bc5-44c4-97ed-068232825e36" containerID="9f620f46dca7e8791d502112c19efe3cdadb244d25b8bb4a1f8e00fd516a3a2f" exitCode=0
Oct 29 22:10:38.707364 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.704639    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:38.707364 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.705253    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:38.740687 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:10:38.740638    4962 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.689s"
Oct 29 22:10:38.741044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.741028    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 22:10:38.741423 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.741366    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:10:38.741589 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.741578    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:10:38.741736 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.741725    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 22:10:38.741874 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.741856    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" event={"ID":"f58653d9-6bc5-44c4-97ed-068232825e36","Type":"ContainerDied","Data":"9f620f46dca7e8791d502112c19efe3cdadb244d25b8bb4a1f8e00fd516a3a2f"}
Oct 29 22:10:38.742050 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.742023    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:10:38.742188 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.742178    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 22:10:38.750532 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.748730    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:10:38.764933 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.764897    4962 generic.go:356] "Generic (PLEG): container finished" podID="692ec1aa-ccc0-4712-88bd-b555b8eb9dae" containerID="acdac0b3a761ff0b0b767da39f44cf8b0a77a0c3d758b7d5e4f8735def15f7dd" exitCode=0
Oct 29 22:10:38.799315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.768043    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb" event={"ID":"692ec1aa-ccc0-4712-88bd-b555b8eb9dae","Type":"ContainerDied","Data":"acdac0b3a761ff0b0b767da39f44cf8b0a77a0c3d758b7d5e4f8735def15f7dd"}
Oct 29 22:10:38.857311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:38.822767    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:39.069853 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.068701    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Oct 29 22:10:39.069853 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.068762    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h" podUID="c1d2bd1d-783b-415a-8807-8f53396ec0ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Oct 29 22:10:39.131339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.131310    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:39.135331 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.135304    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-workload-certs\") pod \"f58653d9-6bc5-44c4-97ed-068232825e36\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") "
Oct 29 22:10:39.135591 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.135576    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-credential-socket\") pod \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") "
Oct 29 22:10:39.135715 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.135702    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f58653d9-6bc5-44c4-97ed-068232825e36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58653d9-6bc5-44c4-97ed-068232825e36-istio-podinfo\") pod \"f58653d9-6bc5-44c4-97ed-068232825e36\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") "
Oct 29 22:10:39.135839 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.135825    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-workload-socket\") pod \"f58653d9-6bc5-44c4-97ed-068232825e36\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") "
Oct 29 22:10:39.135976 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.135963    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istiod-ca-cert\") pod \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") "
Oct 29 22:10:39.136102 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.136090    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58653d9-6bc5-44c4-97ed-068232825e36-istio-token\" (UniqueName: \"kubernetes.io/projected/f58653d9-6bc5-44c4-97ed-068232825e36-istio-token\") pod \"f58653d9-6bc5-44c4-97ed-068232825e36\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") "
Oct 29 22:10:39.136193 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.136181    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-istio-envoy\") pod \"f58653d9-6bc5-44c4-97ed-068232825e36\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") "
Oct 29 22:10:39.136314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.136301    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58653d9-6bc5-44c4-97ed-068232825e36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58653d9-6bc5-44c4-97ed-068232825e36-istiod-ca-cert\") pod \"f58653d9-6bc5-44c4-97ed-068232825e36\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") "
Oct 29 22:10:39.136426 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.136413    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-envoy\") pod \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") "
Oct 29 22:10:39.136594 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.136581    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-data\") pod \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") "
Oct 29 22:10:39.136750 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.136736    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58653d9-6bc5-44c4-97ed-068232825e36-kube-api-access-xncl6\" (UniqueName: \"kubernetes.io/projected/f58653d9-6bc5-44c4-97ed-068232825e36-kube-api-access-xncl6\") pod \"f58653d9-6bc5-44c4-97ed-068232825e36\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") "
Oct 29 22:10:39.136869 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.136857    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-podinfo\") pod \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") "
Oct 29 22:10:39.137025 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.137011    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-token\" (UniqueName: \"kubernetes.io/projected/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-token\") pod \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") "
Oct 29 22:10:39.137136 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.137124    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-istio-data\") pod \"f58653d9-6bc5-44c4-97ed-068232825e36\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") "
Oct 29 22:10:39.137392 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.137375    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-workload-certs\") pod \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") "
Oct 29 22:10:39.137556 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.137540    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-workload-socket\") pod \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") "
Oct 29 22:10:39.137701 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.137689    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-credential-socket\") pod \"f58653d9-6bc5-44c4-97ed-068232825e36\" (UID: \"f58653d9-6bc5-44c4-97ed-068232825e36\") "
Oct 29 22:10:39.137837 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.137825    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-kube-api-access-6fndv\" (UniqueName: \"kubernetes.io/projected/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-kube-api-access-6fndv\") pod \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\" (UID: \"692ec1aa-ccc0-4712-88bd-b555b8eb9dae\") "
Oct 29 22:10:39.143250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.143219    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-data" pod "692ec1aa-ccc0-4712-88bd-b555b8eb9dae" (UID: "692ec1aa-ccc0-4712-88bd-b555b8eb9dae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:39.160016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.146861    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-token" pod "692ec1aa-ccc0-4712-88bd-b555b8eb9dae" (UID: "692ec1aa-ccc0-4712-88bd-b555b8eb9dae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:39.162345 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.160255    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-podinfo" pod "692ec1aa-ccc0-4712-88bd-b555b8eb9dae" (UID: "692ec1aa-ccc0-4712-88bd-b555b8eb9dae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:10:39.162345 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.161844    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-workload-certs" pod "f58653d9-6bc5-44c4-97ed-068232825e36" (UID: "f58653d9-6bc5-44c4-97ed-068232825e36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:39.162345 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.161862    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-credential-socket" pod "f58653d9-6bc5-44c4-97ed-068232825e36" (UID: "f58653d9-6bc5-44c4-97ed-068232825e36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:39.162345 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.162031    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-credential-socket" pod "692ec1aa-ccc0-4712-88bd-b555b8eb9dae" (UID: "692ec1aa-ccc0-4712-88bd-b555b8eb9dae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:39.162646 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.162613    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-kube-api-access-6fndv" pod "692ec1aa-ccc0-4712-88bd-b555b8eb9dae" (UID: "692ec1aa-ccc0-4712-88bd-b555b8eb9dae"). InnerVolumeSpecName "kube-api-access-6fndv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:39.163722 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.162767    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-workload-socket" pod "692ec1aa-ccc0-4712-88bd-b555b8eb9dae" (UID: "692ec1aa-ccc0-4712-88bd-b555b8eb9dae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:39.163722 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.163117    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istiod-ca-cert" pod "692ec1aa-ccc0-4712-88bd-b555b8eb9dae" (UID: "692ec1aa-ccc0-4712-88bd-b555b8eb9dae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:39.163722 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.163239    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-workload-socket" pod "f58653d9-6bc5-44c4-97ed-068232825e36" (UID: "f58653d9-6bc5-44c4-97ed-068232825e36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:39.163722 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.163486    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-istio-data" pod "f58653d9-6bc5-44c4-97ed-068232825e36" (UID: "f58653d9-6bc5-44c4-97ed-068232825e36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:39.164662 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.164611    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:10:39.165137 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.165075    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58653d9-6bc5-44c4-97ed-068232825e36-istiod-ca-cert" pod "f58653d9-6bc5-44c4-97ed-068232825e36" (UID: "f58653d9-6bc5-44c4-97ed-068232825e36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:39.165919 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.165468    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-workload-certs" pod "692ec1aa-ccc0-4712-88bd-b555b8eb9dae" (UID: "692ec1aa-ccc0-4712-88bd-b555b8eb9dae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:39.175801 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.175778    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58653d9-6bc5-44c4-97ed-068232825e36-kube-api-access-xncl6" pod "f58653d9-6bc5-44c4-97ed-068232825e36" (UID: "f58653d9-6bc5-44c4-97ed-068232825e36"). InnerVolumeSpecName "kube-api-access-xncl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:39.229728 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.221257    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f58653d9-6bc5-44c4-97ed-068232825e36-istio-podinfo" pod "f58653d9-6bc5-44c4-97ed-068232825e36" (UID: "f58653d9-6bc5-44c4-97ed-068232825e36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:10:39.229936 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.215695    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:39.230110 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.230096    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:10:39.230227 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.230216    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 22:10:39.230369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.230358    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"
Oct 29 22:10:39.230492 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.230479    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"]
Oct 29 22:10:39.230596 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.230587    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-sgfzs"]
Oct 29 22:10:39.233473 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.233453    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:39.234568 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.234531    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-envoy" pod "692ec1aa-ccc0-4712-88bd-b555b8eb9dae" (UID: "692ec1aa-ccc0-4712-88bd-b555b8eb9dae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:39.234677 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.234627    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-istio-envoy" pod "f58653d9-6bc5-44c4-97ed-068232825e36" (UID: "f58653d9-6bc5-44c4-97ed-068232825e36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:39.235802 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.235773    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58653d9-6bc5-44c4-97ed-068232825e36-istio-token" pod "f58653d9-6bc5-44c4-97ed-068232825e36" (UID: "f58653d9-6bc5-44c4-97ed-068232825e36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:39.241416 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.241390    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xncl6\" (UniqueName: \"kubernetes.io/projected/f58653d9-6bc5-44c4-97ed-068232825e36-kube-api-access-xncl6\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:39.241531 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.241420    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:39.241531 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.241436    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:39.241531 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.241450    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:39.241531 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.241463    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:39.241531 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.241475    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:39.241531 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.241490    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:39.241531 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.241504    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fndv\" (UniqueName: \"kubernetes.io/projected/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-kube-api-access-6fndv\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:39.241531 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.241517    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:39.241531 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.241530    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:39.242003 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.241543    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58653d9-6bc5-44c4-97ed-068232825e36-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:39.242003 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.241595    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:39.242003 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.241611    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:39.242003 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.241632    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58653d9-6bc5-44c4-97ed-068232825e36-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:39.242003 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.241646    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58653d9-6bc5-44c4-97ed-068232825e36-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:39.242003 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.241661    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58653d9-6bc5-44c4-97ed-068232825e36-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:39.242003 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.241674    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:39.242003 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.241687    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/692ec1aa-ccc0-4712-88bd-b555b8eb9dae-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:39.248610 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.248501    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 22:10:39.252088 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.251638    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:10:39.520453 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.520338    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:39.523994 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.523833    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:39.525616 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.525569    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:10:39.789021 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:10:39.788934    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:10:39.799681 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.799649    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"
Oct 29 22:10:39.799832 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.799771    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb" event={"ID":"692ec1aa-ccc0-4712-88bd-b555b8eb9dae","Type":"ContainerDied","Data":"90679bae151a874ee40433f9ed4f83739df0e481cf6e8b0ed2d5602dfa4acfde"}
Oct 29 22:10:39.799949 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.799852    4962 scope.go:122] "RemoveContainer" containerID="acdac0b3a761ff0b0b767da39f44cf8b0a77a0c3d758b7d5e4f8735def15f7dd"
Oct 29 22:10:39.803361 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.803321    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"
Oct 29 22:10:39.810645 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.810620    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt" event={"ID":"f58653d9-6bc5-44c4-97ed-068232825e36","Type":"ContainerDied","Data":"49e79333805473445908f9bd94cee5ecf7f8eef0416235090a7e3c1fa66b8b05"}
Oct 29 22:10:39.817249 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.817225    4962 scope.go:122] "RemoveContainer" containerID="9f620f46dca7e8791d502112c19efe3cdadb244d25b8bb4a1f8e00fd516a3a2f"
Oct 29 22:10:39.955842 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.955802    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"]
Oct 29 22:10:39.962816 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.962569    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-8lcvb"]
Oct 29 22:10:39.964683 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.964662    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:39.965190 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.965168    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:39.965809 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.965787    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:10:39.982940 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.982915    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"]
Oct 29 22:10:39.988015 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:39.987994    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-df42c7-openshift-default-6bc46b48f-d2mpt"]
Oct 29 22:10:41.065064 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:41.065023    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1693a089-535d-4f1d-815b-075639d9b9ab" path="/var/lib/kubelet/pods/1693a089-535d-4f1d-815b-075639d9b9ab/volumes"
Oct 29 22:10:41.065981 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:41.065956    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="692ec1aa-ccc0-4712-88bd-b555b8eb9dae" path="/var/lib/kubelet/pods/692ec1aa-ccc0-4712-88bd-b555b8eb9dae/volumes"
Oct 29 22:10:41.066901 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:41.066875    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" path="/var/lib/kubelet/pods/f58653d9-6bc5-44c4-97ed-068232825e36/volumes"
Oct 29 22:10:41.329978 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:41.329901    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z"]
Oct 29 22:10:42.474471 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:42.470800    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" containerID="cri-o://b3559de0ac25a5d498778e4412fa0d991baf71793ec3ed0a09fcda072634e633" gracePeriod=35
Oct 29 22:10:42.629067 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:42.629021    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": dial tcp 10.132.0.72:8000: connect: connection refused"
Oct 29 22:10:42.928315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:42.925805    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 22:10:42.966921 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:42.965116    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-home\" (UniqueName: \"kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-home\") pod \"f1ebf370-2322-49c9-8615-2d7109f57083\" (UID: \"f1ebf370-2322-49c9-8615-2d7109f57083\") "
Oct 29 22:10:42.966921 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:42.965249    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ebf370-2322-49c9-8615-2d7109f57083-kube-api-access-7l7pn\" (UniqueName: \"kubernetes.io/projected/f1ebf370-2322-49c9-8615-2d7109f57083-kube-api-access-7l7pn\") pod \"f1ebf370-2322-49c9-8615-2d7109f57083\" (UID: \"f1ebf370-2322-49c9-8615-2d7109f57083\") "
Oct 29 22:10:42.966921 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:42.965395    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-dshm\") pod \"f1ebf370-2322-49c9-8615-2d7109f57083\" (UID: \"f1ebf370-2322-49c9-8615-2d7109f57083\") "
Oct 29 22:10:42.966921 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:42.965478    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-tmp-dir\") pod \"f1ebf370-2322-49c9-8615-2d7109f57083\" (UID: \"f1ebf370-2322-49c9-8615-2d7109f57083\") "
Oct 29 22:10:42.966921 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:42.965654    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1ebf370-2322-49c9-8615-2d7109f57083-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1ebf370-2322-49c9-8615-2d7109f57083-tls-certs\") pod \"f1ebf370-2322-49c9-8615-2d7109f57083\" (UID: \"f1ebf370-2322-49c9-8615-2d7109f57083\") "
Oct 29 22:10:42.966921 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:42.965773    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-model-cache\") pod \"f1ebf370-2322-49c9-8615-2d7109f57083\" (UID: \"f1ebf370-2322-49c9-8615-2d7109f57083\") "
Oct 29 22:10:42.966921 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:42.966579    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-model-cache" pod "f1ebf370-2322-49c9-8615-2d7109f57083" (UID: "f1ebf370-2322-49c9-8615-2d7109f57083"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:42.966921 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:42.966833    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-tmp-dir" pod "f1ebf370-2322-49c9-8615-2d7109f57083" (UID: "f1ebf370-2322-49c9-8615-2d7109f57083"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:42.966921 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:42.966905    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-home" pod "f1ebf370-2322-49c9-8615-2d7109f57083" (UID: "f1ebf370-2322-49c9-8615-2d7109f57083"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:42.968418 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:42.968342    4962 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-model-cache\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:42.968418 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:42.968370    4962 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-home\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:42.968418 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:42.968383    4962 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-tmp-dir\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:42.971254 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:42.971222    4962 generic.go:356] "Generic (PLEG): container finished" podID="f1ebf370-2322-49c9-8615-2d7109f57083" containerID="b3559de0ac25a5d498778e4412fa0d991baf71793ec3ed0a09fcda072634e633" exitCode=0
Oct 29 22:10:42.971410 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:42.971339    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" event={"ID":"f1ebf370-2322-49c9-8615-2d7109f57083","Type":"ContainerDied","Data":"b3559de0ac25a5d498778e4412fa0d991baf71793ec3ed0a09fcda072634e633"}
Oct 29 22:10:42.971410 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:42.971380    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd" event={"ID":"f1ebf370-2322-49c9-8615-2d7109f57083","Type":"ContainerDied","Data":"790109bbc7971f3f1275bbfd62df875c0c35e8f64aa031164bf936cfd336e0fb"}
Oct 29 22:10:42.971410 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:42.971403    4962 scope.go:122] "RemoveContainer" containerID="b3559de0ac25a5d498778e4412fa0d991baf71793ec3ed0a09fcda072634e633"
Oct 29 22:10:42.971579 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:42.971559    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"
Oct 29 22:10:42.972786 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:42.972625    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1ebf370-2322-49c9-8615-2d7109f57083-tls-certs" pod "f1ebf370-2322-49c9-8615-2d7109f57083" (UID: "f1ebf370-2322-49c9-8615-2d7109f57083"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:10:43.101241 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:43.101170    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ebf370-2322-49c9-8615-2d7109f57083-kube-api-access-7l7pn" pod "f1ebf370-2322-49c9-8615-2d7109f57083" (UID: "f1ebf370-2322-49c9-8615-2d7109f57083"). InnerVolumeSpecName "kube-api-access-7l7pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:43.110790 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:43.106678    4962 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1ebf370-2322-49c9-8615-2d7109f57083-tls-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:43.175902 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:43.175760    4962 scope.go:122] "RemoveContainer" containerID="b3559de0ac25a5d498778e4412fa0d991baf71793ec3ed0a09fcda072634e633"
Oct 29 22:10:43.176856 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:10:43.176765    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3559de0ac25a5d498778e4412fa0d991baf71793ec3ed0a09fcda072634e633\": container with ID starting with b3559de0ac25a5d498778e4412fa0d991baf71793ec3ed0a09fcda072634e633 not found: ID does not exist" containerID="b3559de0ac25a5d498778e4412fa0d991baf71793ec3ed0a09fcda072634e633"
Oct 29 22:10:43.176856 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:43.176808    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3559de0ac25a5d498778e4412fa0d991baf71793ec3ed0a09fcda072634e633"} err="failed to get container status \"b3559de0ac25a5d498778e4412fa0d991baf71793ec3ed0a09fcda072634e633\": rpc error: code = NotFound desc = could not find container \"b3559de0ac25a5d498778e4412fa0d991baf71793ec3ed0a09fcda072634e633\": container with ID starting with b3559de0ac25a5d498778e4412fa0d991baf71793ec3ed0a09fcda072634e633 not found: ID does not exist"
Oct 29 22:10:43.202924 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:43.201866    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-dshm" pod "f1ebf370-2322-49c9-8615-2d7109f57083" (UID: "f1ebf370-2322-49c9-8615-2d7109f57083"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:43.208963 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:43.208933    4962 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1ebf370-2322-49c9-8615-2d7109f57083-dshm\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:43.209128 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:43.209115    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l7pn\" (UniqueName: \"kubernetes.io/projected/f1ebf370-2322-49c9-8615-2d7109f57083-kube-api-access-7l7pn\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:43.338771 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:43.338694    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"]
Oct 29 22:10:43.350553 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:43.350524    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a9def7/simulator-w2-a9def7-kserve-5ff46c6f67-kvldd"]
Oct 29 22:10:43.991577 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:43.991548    4962 generic.go:356] "Generic (PLEG): container finished" podID="bced75a3-37ba-4595-9dc0-b354574eba07" containerID="5ab3ad1e4ae32a29b221951ad5682083847f64fb401f6a51f2a83afd4d835689" exitCode=0
Oct 29 22:10:43.992217 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:43.991621    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn" event={"ID":"bced75a3-37ba-4595-9dc0-b354574eba07","Type":"ContainerDied","Data":"5ab3ad1e4ae32a29b221951ad5682083847f64fb401f6a51f2a83afd4d835689"}
Oct 29 22:10:43.992217 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:43.992069    4962 scope.go:122] "RemoveContainer" containerID="5ab3ad1e4ae32a29b221951ad5682083847f64fb401f6a51f2a83afd4d835689"
Oct 29 22:10:43.993798 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:43.993765    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:10:43.993912 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:43.993856    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:10:43.993912 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:43.993887    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:10:45.007058 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:45.007022    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn" event={"ID":"bced75a3-37ba-4595-9dc0-b354574eba07","Type":"ContainerStarted","Data":"678eac811730ab90c261ea74bfe16e0c5ed7e0a918d54786252bd39a6f922e73"}
Oct 29 22:10:45.060176 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:45.060148    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" path="/var/lib/kubelet/pods/f1ebf370-2322-49c9-8615-2d7109f57083/volumes"
Oct 29 22:10:45.612545 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:45.612513    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:10:45.613001 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:45.612978    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 29 22:10:45.613053 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:45.613036    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn" podUID="bced75a3-37ba-4595-9dc0-b354574eba07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 29 22:10:46.330992 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:46.330950    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z" podUID="b64e9d74-639d-44be-bdd1-04c9ef4f81fc" containerName="payload-pre-processing" containerID="cri-o://c6190725a37b65115a3918d8f4bedf712e96ac58f884f2958fe990d0ee9dfaa5" gracePeriod=25
Oct 29 22:10:46.431281 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:46.431249    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:10:46.612599 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:46.612515    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 29 22:10:46.612599 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:46.612579    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn" podUID="bced75a3-37ba-4595-9dc0-b354574eba07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 29 22:10:47.164946 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:47.164910    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z" podUID="b64e9d74-639d-44be-bdd1-04c9ef4f81fc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:10:47.613170 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:47.613089    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 29 22:10:47.613170 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:47.613155    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn" podUID="bced75a3-37ba-4595-9dc0-b354574eba07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 29 22:10:48.613043 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:48.612997    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 29 22:10:48.613218 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:48.613073    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn" podUID="bced75a3-37ba-4595-9dc0-b354574eba07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 29 22:10:49.613313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:49.612824    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 29 22:10:49.613313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:49.612892    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn" podUID="bced75a3-37ba-4595-9dc0-b354574eba07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 29 22:10:50.680004 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:50.669655    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 29 22:10:50.692305 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:50.692244    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn" podUID="bced75a3-37ba-4595-9dc0-b354574eba07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 29 22:10:51.511352 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:51.510165    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch"]
Oct 29 22:10:51.515133 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:51.515078    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg"]
Oct 29 22:10:51.621094 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:51.621065    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:10:51.622354 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:51.622333    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:10:52.521400 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:52.521353    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main" containerID="cri-o://dc0df5cada975a571e11ca34856ddd616cb7649ab2f45b080b8806380b5e66c3" gracePeriod=41
Oct 29 22:10:52.845885 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:52.845857    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:10:52.914344 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:52.914273    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-dshm\") pod \"fbcad86b-328d-49a0-a993-63dc68f154a4\" (UID: \"fbcad86b-328d-49a0-a993-63dc68f154a4\") "
Oct 29 22:10:52.914496 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:52.914485    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-model-cache\") pod \"fbcad86b-328d-49a0-a993-63dc68f154a4\" (UID: \"fbcad86b-328d-49a0-a993-63dc68f154a4\") "
Oct 29 22:10:52.914539 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:52.914510    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-home\" (UniqueName: \"kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-home\") pod \"fbcad86b-328d-49a0-a993-63dc68f154a4\" (UID: \"fbcad86b-328d-49a0-a993-63dc68f154a4\") "
Oct 29 22:10:52.914589 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:52.914542    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbcad86b-328d-49a0-a993-63dc68f154a4-tls-certs\" (UniqueName: \"kubernetes.io/secret/fbcad86b-328d-49a0-a993-63dc68f154a4-tls-certs\") pod \"fbcad86b-328d-49a0-a993-63dc68f154a4\" (UID: \"fbcad86b-328d-49a0-a993-63dc68f154a4\") "
Oct 29 22:10:52.914667 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:52.914651    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-tmp-dir\") pod \"fbcad86b-328d-49a0-a993-63dc68f154a4\" (UID: \"fbcad86b-328d-49a0-a993-63dc68f154a4\") "
Oct 29 22:10:52.914803 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:52.914785    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbcad86b-328d-49a0-a993-63dc68f154a4-kube-api-access-t56t8\" (UniqueName: \"kubernetes.io/projected/fbcad86b-328d-49a0-a993-63dc68f154a4-kube-api-access-t56t8\") pod \"fbcad86b-328d-49a0-a993-63dc68f154a4\" (UID: \"fbcad86b-328d-49a0-a993-63dc68f154a4\") "
Oct 29 22:10:52.914868 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:52.914792    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-model-cache" pod "fbcad86b-328d-49a0-a993-63dc68f154a4" (UID: "fbcad86b-328d-49a0-a993-63dc68f154a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:52.914868 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:52.914799    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-home" pod "fbcad86b-328d-49a0-a993-63dc68f154a4" (UID: "fbcad86b-328d-49a0-a993-63dc68f154a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:52.915024 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:52.915003    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-tmp-dir" pod "fbcad86b-328d-49a0-a993-63dc68f154a4" (UID: "fbcad86b-328d-49a0-a993-63dc68f154a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:52.915663 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:52.915642    4962 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-model-cache\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:52.915762 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:52.915665    4962 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-home\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:52.915762 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:52.915678    4962 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-tmp-dir\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:52.917477 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:52.917451    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-dshm" pod "fbcad86b-328d-49a0-a993-63dc68f154a4" (UID: "fbcad86b-328d-49a0-a993-63dc68f154a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:10:52.917562 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:52.917520    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbcad86b-328d-49a0-a993-63dc68f154a4-kube-api-access-t56t8" pod "fbcad86b-328d-49a0-a993-63dc68f154a4" (UID: "fbcad86b-328d-49a0-a993-63dc68f154a4"). InnerVolumeSpecName "kube-api-access-t56t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:52.917562 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:52.917533    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbcad86b-328d-49a0-a993-63dc68f154a4-tls-certs" pod "fbcad86b-328d-49a0-a993-63dc68f154a4" (UID: "fbcad86b-328d-49a0-a993-63dc68f154a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:10:53.017653 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:53.017622    4962 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbcad86b-328d-49a0-a993-63dc68f154a4-dshm\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:53.017653 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:53.017649    4962 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbcad86b-328d-49a0-a993-63dc68f154a4-tls-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:53.017653 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:53.017660    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t56t8\" (UniqueName: \"kubernetes.io/projected/fbcad86b-328d-49a0-a993-63dc68f154a4-kube-api-access-t56t8\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:53.518937 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:53.518898    4962 generic.go:356] "Generic (PLEG): container finished" podID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerID="dc0df5cada975a571e11ca34856ddd616cb7649ab2f45b080b8806380b5e66c3" exitCode=0
Oct 29 22:10:53.519103 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:53.518939    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"
Oct 29 22:10:53.519103 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:53.518981    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" event={"ID":"fbcad86b-328d-49a0-a993-63dc68f154a4","Type":"ContainerDied","Data":"dc0df5cada975a571e11ca34856ddd616cb7649ab2f45b080b8806380b5e66c3"}
Oct 29 22:10:53.519103 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:53.519026    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk" event={"ID":"fbcad86b-328d-49a0-a993-63dc68f154a4","Type":"ContainerDied","Data":"4b2c60272b608331d3ff1552c9a479351970c5f699ec0a53f01a21ca7950d70d"}
Oct 29 22:10:53.519103 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:53.519045    4962 scope.go:122] "RemoveContainer" containerID="dc0df5cada975a571e11ca34856ddd616cb7649ab2f45b080b8806380b5e66c3"
Oct 29 22:10:53.530203 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:53.529986    4962 scope.go:122] "RemoveContainer" containerID="dc0df5cada975a571e11ca34856ddd616cb7649ab2f45b080b8806380b5e66c3"
Oct 29 22:10:53.530579 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:10:53.530240    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc0df5cada975a571e11ca34856ddd616cb7649ab2f45b080b8806380b5e66c3\": container with ID starting with dc0df5cada975a571e11ca34856ddd616cb7649ab2f45b080b8806380b5e66c3 not found: ID does not exist" containerID="dc0df5cada975a571e11ca34856ddd616cb7649ab2f45b080b8806380b5e66c3"
Oct 29 22:10:53.530579 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:53.530264    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc0df5cada975a571e11ca34856ddd616cb7649ab2f45b080b8806380b5e66c3"} err="failed to get container status \"dc0df5cada975a571e11ca34856ddd616cb7649ab2f45b080b8806380b5e66c3\": rpc error: code = NotFound desc = could not find container \"dc0df5cada975a571e11ca34856ddd616cb7649ab2f45b080b8806380b5e66c3\": container with ID starting with dc0df5cada975a571e11ca34856ddd616cb7649ab2f45b080b8806380b5e66c3 not found: ID does not exist"
Oct 29 22:10:53.575014 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:53.574980    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"]
Oct 29 22:10:53.585961 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:53.585933    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a3c02e/distinct-2-w1-a3c02e-kserve-6d7945fd85-zblkk"]
Oct 29 22:10:55.063814 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:55.063784    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" path="/var/lib/kubelet/pods/fbcad86b-328d-49a0-a993-63dc68f154a4/volumes"
Oct 29 22:10:56.510890 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:56.510849    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch" podUID="5c4bfc57-2b32-410a-ab06-610e5749ab0e" containerName="payload-processing" containerID="cri-o://9f42c8ab44f21d74130238a4f1014bc99f670c8c66a6cd6aab9c41a49f0dc0f4" gracePeriod=25
Oct 29 22:10:56.516379 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:56.516345    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg" podUID="f592a322-7cc5-41d0-8212-18797b23b843" containerName="payload-pre-processing" containerID="cri-o://827840f7af12ff8add91cbb0dd1043bbb2000669b85a5a5d00904d51aa2d23d6" gracePeriod=25
Oct 29 22:10:56.572681 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:10:56.572662    4962 logging.go:55] [core] [Channel #3301 SubChannel #3302]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.151:50052", ServerName: "10.132.0.151:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.151:50052: connect: connection refused"
Oct 29 22:10:56.765948 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:56.764487    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5"]
Oct 29 22:10:56.824411 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:56.824385    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch"
Oct 29 22:10:56.915108 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:56.915078    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg"
Oct 29 22:10:57.018541 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.018458    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f592a322-7cc5-41d0-8212-18797b23b843-kube-api-access-gwtz6\" (UniqueName: \"kubernetes.io/projected/f592a322-7cc5-41d0-8212-18797b23b843-kube-api-access-gwtz6\") pod \"f592a322-7cc5-41d0-8212-18797b23b843\" (UID: \"f592a322-7cc5-41d0-8212-18797b23b843\") "
Oct 29 22:10:57.018541 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.018519    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f592a322-7cc5-41d0-8212-18797b23b843-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f592a322-7cc5-41d0-8212-18797b23b843-plugins-config-volume\") pod \"f592a322-7cc5-41d0-8212-18797b23b843\" (UID: \"f592a322-7cc5-41d0-8212-18797b23b843\") "
Oct 29 22:10:57.018830 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.018811    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c4bfc57-2b32-410a-ab06-610e5749ab0e-kube-api-access-zssk7\" (UniqueName: \"kubernetes.io/projected/5c4bfc57-2b32-410a-ab06-610e5749ab0e-kube-api-access-zssk7\") pod \"5c4bfc57-2b32-410a-ab06-610e5749ab0e\" (UID: \"5c4bfc57-2b32-410a-ab06-610e5749ab0e\") "
Oct 29 22:10:57.018928 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.018883    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f592a322-7cc5-41d0-8212-18797b23b843-plugins-config-volume" pod "f592a322-7cc5-41d0-8212-18797b23b843" (UID: "f592a322-7cc5-41d0-8212-18797b23b843"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:57.018992 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.018938    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c4bfc57-2b32-410a-ab06-610e5749ab0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c4bfc57-2b32-410a-ab06-610e5749ab0e-plugins-config-volume\") pod \"5c4bfc57-2b32-410a-ab06-610e5749ab0e\" (UID: \"5c4bfc57-2b32-410a-ab06-610e5749ab0e\") "
Oct 29 22:10:57.019363 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.019338    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c4bfc57-2b32-410a-ab06-610e5749ab0e-plugins-config-volume" pod "5c4bfc57-2b32-410a-ab06-610e5749ab0e" (UID: "5c4bfc57-2b32-410a-ab06-610e5749ab0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:10:57.019770 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.019751    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c4bfc57-2b32-410a-ab06-610e5749ab0e-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:57.019832 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.019775    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f592a322-7cc5-41d0-8212-18797b23b843-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:57.020769 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.020742    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f592a322-7cc5-41d0-8212-18797b23b843-kube-api-access-gwtz6" pod "f592a322-7cc5-41d0-8212-18797b23b843" (UID: "f592a322-7cc5-41d0-8212-18797b23b843"). InnerVolumeSpecName "kube-api-access-gwtz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:57.020897 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.020868    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c4bfc57-2b32-410a-ab06-610e5749ab0e-kube-api-access-zssk7" pod "5c4bfc57-2b32-410a-ab06-610e5749ab0e" (UID: "5c4bfc57-2b32-410a-ab06-610e5749ab0e"). InnerVolumeSpecName "kube-api-access-zssk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:10:57.148037 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.126514    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zssk7\" (UniqueName: \"kubernetes.io/projected/5c4bfc57-2b32-410a-ab06-610e5749ab0e-kube-api-access-zssk7\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:57.155246 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.155206    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwtz6\" (UniqueName: \"kubernetes.io/projected/f592a322-7cc5-41d0-8212-18797b23b843-kube-api-access-gwtz6\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:10:57.192905 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.192458    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z" podUID="b64e9d74-639d-44be-bdd1-04c9ef4f81fc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:10:57.588420 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.588242    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg" podUID="f592a322-7cc5-41d0-8212-18797b23b843" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.151:50052\" within 1s: context deadline exceeded"
Oct 29 22:10:57.657202 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.657158    4962 generic.go:356] "Generic (PLEG): container finished" podID="5c4bfc57-2b32-410a-ab06-610e5749ab0e" containerID="9f42c8ab44f21d74130238a4f1014bc99f670c8c66a6cd6aab9c41a49f0dc0f4" exitCode=0
Oct 29 22:10:57.657635 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.657606    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch" event={"ID":"5c4bfc57-2b32-410a-ab06-610e5749ab0e","Type":"ContainerDied","Data":"9f42c8ab44f21d74130238a4f1014bc99f670c8c66a6cd6aab9c41a49f0dc0f4"}
Oct 29 22:10:57.657898 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.657876    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch" event={"ID":"5c4bfc57-2b32-410a-ab06-610e5749ab0e","Type":"ContainerDied","Data":"3238db60a092576f5b28733ab84d1d46865f97765f22214a7c92cab1f90ed958"}
Oct 29 22:10:57.662508 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.658426    4962 scope.go:122] "RemoveContainer" containerID="9f42c8ab44f21d74130238a4f1014bc99f670c8c66a6cd6aab9c41a49f0dc0f4"
Oct 29 22:10:57.662508 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.658668    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch"
Oct 29 22:10:57.710308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.690641    4962 generic.go:356] "Generic (PLEG): container finished" podID="f592a322-7cc5-41d0-8212-18797b23b843" containerID="827840f7af12ff8add91cbb0dd1043bbb2000669b85a5a5d00904d51aa2d23d6" exitCode=0
Oct 29 22:10:57.710308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.690756    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg" event={"ID":"f592a322-7cc5-41d0-8212-18797b23b843","Type":"ContainerDied","Data":"827840f7af12ff8add91cbb0dd1043bbb2000669b85a5a5d00904d51aa2d23d6"}
Oct 29 22:10:57.710308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.690826    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg" event={"ID":"f592a322-7cc5-41d0-8212-18797b23b843","Type":"ContainerDied","Data":"21d6050e1fe01b05bc01e9a006ac88420dedb3722ebb72e63f733d7b31da3878"}
Oct 29 22:10:57.710308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.690954    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg"
Oct 29 22:10:57.840646 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.840602    4962 scope.go:122] "RemoveContainer" containerID="9f42c8ab44f21d74130238a4f1014bc99f670c8c66a6cd6aab9c41a49f0dc0f4"
Oct 29 22:10:57.841177 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:10:57.841146    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f42c8ab44f21d74130238a4f1014bc99f670c8c66a6cd6aab9c41a49f0dc0f4\": container with ID starting with 9f42c8ab44f21d74130238a4f1014bc99f670c8c66a6cd6aab9c41a49f0dc0f4 not found: ID does not exist" containerID="9f42c8ab44f21d74130238a4f1014bc99f670c8c66a6cd6aab9c41a49f0dc0f4"
Oct 29 22:10:57.841449 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.841189    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f42c8ab44f21d74130238a4f1014bc99f670c8c66a6cd6aab9c41a49f0dc0f4"} err="failed to get container status \"9f42c8ab44f21d74130238a4f1014bc99f670c8c66a6cd6aab9c41a49f0dc0f4\": rpc error: code = NotFound desc = could not find container \"9f42c8ab44f21d74130238a4f1014bc99f670c8c66a6cd6aab9c41a49f0dc0f4\": container with ID starting with 9f42c8ab44f21d74130238a4f1014bc99f670c8c66a6cd6aab9c41a49f0dc0f4 not found: ID does not exist"
Oct 29 22:10:57.841449 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.841249    4962 scope.go:122] "RemoveContainer" containerID="827840f7af12ff8add91cbb0dd1043bbb2000669b85a5a5d00904d51aa2d23d6"
Oct 29 22:10:57.853345 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.853301    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch"]
Oct 29 22:10:57.856364 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.855530    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f82846-66b974f8c4-8x9ch"]
Oct 29 22:10:57.860568 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.860549    4962 scope.go:122] "RemoveContainer" containerID="827840f7af12ff8add91cbb0dd1043bbb2000669b85a5a5d00904d51aa2d23d6"
Oct 29 22:10:57.860935 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:10:57.860913    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"827840f7af12ff8add91cbb0dd1043bbb2000669b85a5a5d00904d51aa2d23d6\": container with ID starting with 827840f7af12ff8add91cbb0dd1043bbb2000669b85a5a5d00904d51aa2d23d6 not found: ID does not exist" containerID="827840f7af12ff8add91cbb0dd1043bbb2000669b85a5a5d00904d51aa2d23d6"
Oct 29 22:10:57.861025 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.860942    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"827840f7af12ff8add91cbb0dd1043bbb2000669b85a5a5d00904d51aa2d23d6"} err="failed to get container status \"827840f7af12ff8add91cbb0dd1043bbb2000669b85a5a5d00904d51aa2d23d6\": rpc error: code = NotFound desc = could not find container \"827840f7af12ff8add91cbb0dd1043bbb2000669b85a5a5d00904d51aa2d23d6\": container with ID starting with 827840f7af12ff8add91cbb0dd1043bbb2000669b85a5a5d00904d51aa2d23d6 not found: ID does not exist"
Oct 29 22:10:57.879376 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.879349    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg"]
Oct 29 22:10:57.880931 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:57.880908    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-54b554d55f-hwzgg"]
Oct 29 22:10:59.059993 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:59.059958    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c4bfc57-2b32-410a-ab06-610e5749ab0e" path="/var/lib/kubelet/pods/5c4bfc57-2b32-410a-ab06-610e5749ab0e/volumes"
Oct 29 22:10:59.060493 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:10:59.060476    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f592a322-7cc5-41d0-8212-18797b23b843" path="/var/lib/kubelet/pods/f592a322-7cc5-41d0-8212-18797b23b843/volumes"
Oct 29 22:11:01.765123 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:01.765087    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5" podUID="f06268ed-ff58-4564-abd0-54609575a363" containerName="payload-pre-processing" containerID="cri-o://cfd30bf342c789cbaaab0bcc04f8826bd668b7d035f28867c1bb5909574cea03" gracePeriod=25
Oct 29 22:11:02.102998 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:02.102971    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5"
Oct 29 22:11:02.134423 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:02.134388    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f06268ed-ff58-4564-abd0-54609575a363-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06268ed-ff58-4564-abd0-54609575a363-plugins-config-volume\") pod \"f06268ed-ff58-4564-abd0-54609575a363\" (UID: \"f06268ed-ff58-4564-abd0-54609575a363\") "
Oct 29 22:11:02.134713 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:02.134697    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f06268ed-ff58-4564-abd0-54609575a363-kube-api-access-4t2lc\" (UniqueName: \"kubernetes.io/projected/f06268ed-ff58-4564-abd0-54609575a363-kube-api-access-4t2lc\") pod \"f06268ed-ff58-4564-abd0-54609575a363\" (UID: \"f06268ed-ff58-4564-abd0-54609575a363\") "
Oct 29 22:11:02.134816 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:02.134787    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f06268ed-ff58-4564-abd0-54609575a363-plugins-config-volume" pod "f06268ed-ff58-4564-abd0-54609575a363" (UID: "f06268ed-ff58-4564-abd0-54609575a363"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:02.135333 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:02.135314    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06268ed-ff58-4564-abd0-54609575a363-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:02.136963 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:02.136940    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06268ed-ff58-4564-abd0-54609575a363-kube-api-access-4t2lc" pod "f06268ed-ff58-4564-abd0-54609575a363" (UID: "f06268ed-ff58-4564-abd0-54609575a363"). InnerVolumeSpecName "kube-api-access-4t2lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:02.236798 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:02.236765    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t2lc\" (UniqueName: \"kubernetes.io/projected/f06268ed-ff58-4564-abd0-54609575a363-kube-api-access-4t2lc\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:02.783832 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:02.783802    4962 generic.go:356] "Generic (PLEG): container finished" podID="f06268ed-ff58-4564-abd0-54609575a363" containerID="cfd30bf342c789cbaaab0bcc04f8826bd668b7d035f28867c1bb5909574cea03" exitCode=0
Oct 29 22:11:02.784311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:02.783867    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5"
Oct 29 22:11:02.784311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:02.783893    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5" event={"ID":"f06268ed-ff58-4564-abd0-54609575a363","Type":"ContainerDied","Data":"cfd30bf342c789cbaaab0bcc04f8826bd668b7d035f28867c1bb5909574cea03"}
Oct 29 22:11:02.784311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:02.783961    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5" event={"ID":"f06268ed-ff58-4564-abd0-54609575a363","Type":"ContainerDied","Data":"9c19a427d89593d51400bfb7480ae6c4381a8939a6eed2178ed4327c8e57fb79"}
Oct 29 22:11:02.784311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:02.783996    4962 scope.go:122] "RemoveContainer" containerID="cfd30bf342c789cbaaab0bcc04f8826bd668b7d035f28867c1bb5909574cea03"
Oct 29 22:11:02.801280 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:02.801260    4962 scope.go:122] "RemoveContainer" containerID="cfd30bf342c789cbaaab0bcc04f8826bd668b7d035f28867c1bb5909574cea03"
Oct 29 22:11:02.801600 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:11:02.801579    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfd30bf342c789cbaaab0bcc04f8826bd668b7d035f28867c1bb5909574cea03\": container with ID starting with cfd30bf342c789cbaaab0bcc04f8826bd668b7d035f28867c1bb5909574cea03 not found: ID does not exist" containerID="cfd30bf342c789cbaaab0bcc04f8826bd668b7d035f28867c1bb5909574cea03"
Oct 29 22:11:02.801655 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:02.801608    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd30bf342c789cbaaab0bcc04f8826bd668b7d035f28867c1bb5909574cea03"} err="failed to get container status \"cfd30bf342c789cbaaab0bcc04f8826bd668b7d035f28867c1bb5909574cea03\": rpc error: code = NotFound desc = could not find container \"cfd30bf342c789cbaaab0bcc04f8826bd668b7d035f28867c1bb5909574cea03\": container with ID starting with cfd30bf342c789cbaaab0bcc04f8826bd668b7d035f28867c1bb5909574cea03 not found: ID does not exist"
Oct 29 22:11:02.835573 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:02.835541    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5"]
Oct 29 22:11:02.846960 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:02.846933    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b80d5b-dfffc75c9-nmfd5"]
Oct 29 22:11:03.060233 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:03.060159    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f06268ed-ff58-4564-abd0-54609575a363" path="/var/lib/kubelet/pods/f06268ed-ff58-4564-abd0-54609575a363/volumes"
Oct 29 22:11:06.385583 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:11:06.385548    4962 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64e9d74_639d_44be_bdd1_04c9ef4f81fc.slice/crio-c6190725a37b65115a3918d8f4bedf712e96ac58f884f2958fe990d0ee9dfaa5.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:11:06.386839 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:11:06.385598    4962 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64e9d74_639d_44be_bdd1_04c9ef4f81fc.slice/crio-c6190725a37b65115a3918d8f4bedf712e96ac58f884f2958fe990d0ee9dfaa5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64e9d74_639d_44be_bdd1_04c9ef4f81fc.slice/crio-conmon-c6190725a37b65115a3918d8f4bedf712e96ac58f884f2958fe990d0ee9dfaa5.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:11:06.629593 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:06.629567    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z"
Oct 29 22:11:06.761942 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:06.761909    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b64e9d74-639d-44be-bdd1-04c9ef4f81fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b64e9d74-639d-44be-bdd1-04c9ef4f81fc-plugins-config-volume\") pod \"b64e9d74-639d-44be-bdd1-04c9ef4f81fc\" (UID: \"b64e9d74-639d-44be-bdd1-04c9ef4f81fc\") "
Oct 29 22:11:06.761942 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:06.761945    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b64e9d74-639d-44be-bdd1-04c9ef4f81fc-kube-api-access-rcs7k\" (UniqueName: \"kubernetes.io/projected/b64e9d74-639d-44be-bdd1-04c9ef4f81fc-kube-api-access-rcs7k\") pod \"b64e9d74-639d-44be-bdd1-04c9ef4f81fc\" (UID: \"b64e9d74-639d-44be-bdd1-04c9ef4f81fc\") "
Oct 29 22:11:06.762304 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:06.762264    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b64e9d74-639d-44be-bdd1-04c9ef4f81fc-plugins-config-volume" pod "b64e9d74-639d-44be-bdd1-04c9ef4f81fc" (UID: "b64e9d74-639d-44be-bdd1-04c9ef4f81fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:06.762956 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:06.762936    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b64e9d74-639d-44be-bdd1-04c9ef4f81fc-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:06.764147 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:06.764122    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b64e9d74-639d-44be-bdd1-04c9ef4f81fc-kube-api-access-rcs7k" pod "b64e9d74-639d-44be-bdd1-04c9ef4f81fc" (UID: "b64e9d74-639d-44be-bdd1-04c9ef4f81fc"). InnerVolumeSpecName "kube-api-access-rcs7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:06.830249 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:06.830219    4962 generic.go:356] "Generic (PLEG): container finished" podID="b64e9d74-639d-44be-bdd1-04c9ef4f81fc" containerID="c6190725a37b65115a3918d8f4bedf712e96ac58f884f2958fe990d0ee9dfaa5" exitCode=0
Oct 29 22:11:06.830383 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:06.830268    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z"
Oct 29 22:11:06.830383 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:06.830309    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z" event={"ID":"b64e9d74-639d-44be-bdd1-04c9ef4f81fc","Type":"ContainerDied","Data":"c6190725a37b65115a3918d8f4bedf712e96ac58f884f2958fe990d0ee9dfaa5"}
Oct 29 22:11:06.830470 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:06.830403    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z" event={"ID":"b64e9d74-639d-44be-bdd1-04c9ef4f81fc","Type":"ContainerDied","Data":"19422e77c04e438b39c1f39fd083f0585d37db3f950c566b99082b1996ca5f40"}
Oct 29 22:11:06.830470 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:06.830434    4962 scope.go:122] "RemoveContainer" containerID="c6190725a37b65115a3918d8f4bedf712e96ac58f884f2958fe990d0ee9dfaa5"
Oct 29 22:11:06.844546 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:06.844524    4962 scope.go:122] "RemoveContainer" containerID="c6190725a37b65115a3918d8f4bedf712e96ac58f884f2958fe990d0ee9dfaa5"
Oct 29 22:11:06.844817 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:11:06.844797    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6190725a37b65115a3918d8f4bedf712e96ac58f884f2958fe990d0ee9dfaa5\": container with ID starting with c6190725a37b65115a3918d8f4bedf712e96ac58f884f2958fe990d0ee9dfaa5 not found: ID does not exist" containerID="c6190725a37b65115a3918d8f4bedf712e96ac58f884f2958fe990d0ee9dfaa5"
Oct 29 22:11:06.844894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:06.844829    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6190725a37b65115a3918d8f4bedf712e96ac58f884f2958fe990d0ee9dfaa5"} err="failed to get container status \"c6190725a37b65115a3918d8f4bedf712e96ac58f884f2958fe990d0ee9dfaa5\": rpc error: code = NotFound desc = could not find container \"c6190725a37b65115a3918d8f4bedf712e96ac58f884f2958fe990d0ee9dfaa5\": container with ID starting with c6190725a37b65115a3918d8f4bedf712e96ac58f884f2958fe990d0ee9dfaa5 not found: ID does not exist"
Oct 29 22:11:06.864747 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:06.864724    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcs7k\" (UniqueName: \"kubernetes.io/projected/b64e9d74-639d-44be-bdd1-04c9ef4f81fc-kube-api-access-rcs7k\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:06.882776 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:06.882748    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z"]
Oct 29 22:11:06.886875 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:06.886853    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a9def7-6b4c4f4f6c-tct9z"]
Oct 29 22:11:07.062773 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:07.062742    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b64e9d74-639d-44be-bdd1-04c9ef4f81fc" path="/var/lib/kubelet/pods/b64e9d74-639d-44be-bdd1-04c9ef4f81fc/volumes"
Oct 29 22:11:09.730594 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:11:09.730554    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:11:11.921720 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.921684    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"]
Oct 29 22:11:11.922953 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.922923    4962 cpu_manager.go:414] "containerMap: removing container" podUID="f592a322-7cc5-41d0-8212-18797b23b843" containerName="payload-pre-processing"
Oct 29 22:11:11.922953 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.922948    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f592a322-7cc5-41d0-8212-18797b23b843" containerName="payload-pre-processing"
Oct 29 22:11:11.923165 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.922976    4962 cpu_manager.go:414] "containerMap: removing container" podUID="5c4bfc57-2b32-410a-ab06-610e5749ab0e" containerName="payload-processing"
Oct 29 22:11:11.923165 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.922985    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c4bfc57-2b32-410a-ab06-610e5749ab0e" containerName="payload-processing"
Oct 29 22:11:11.923165 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.922997    4962 cpu_manager.go:414] "containerMap: removing container" podUID="1693a089-535d-4f1d-815b-075639d9b9ab" containerName="istio-proxy"
Oct 29 22:11:11.923165 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.923004    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1693a089-535d-4f1d-815b-075639d9b9ab" containerName="istio-proxy"
Oct 29 22:11:11.923165 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.923019    4962 cpu_manager.go:414] "containerMap: removing container" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main"
Oct 29 22:11:11.923165 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.923026    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main"
Oct 29 22:11:11.923165 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.923041    4962 cpu_manager.go:414] "containerMap: removing container" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main"
Oct 29 22:11:11.923165 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.923048    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main"
Oct 29 22:11:11.923165 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.923077    4962 cpu_manager.go:414] "containerMap: removing container" podUID="f06268ed-ff58-4564-abd0-54609575a363" containerName="payload-pre-processing"
Oct 29 22:11:11.923165 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.923086    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06268ed-ff58-4564-abd0-54609575a363" containerName="payload-pre-processing"
Oct 29 22:11:11.923165 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.923105    4962 cpu_manager.go:414] "containerMap: removing container" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy"
Oct 29 22:11:11.923165 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.923112    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy"
Oct 29 22:11:11.923165 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.923145    4962 cpu_manager.go:414] "containerMap: removing container" podUID="b64e9d74-639d-44be-bdd1-04c9ef4f81fc" containerName="payload-pre-processing"
Oct 29 22:11:11.923165 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.923152    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b64e9d74-639d-44be-bdd1-04c9ef4f81fc" containerName="payload-pre-processing"
Oct 29 22:11:11.923794 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.923174    4962 cpu_manager.go:414] "containerMap: removing container" podUID="692ec1aa-ccc0-4712-88bd-b555b8eb9dae" containerName="istio-proxy"
Oct 29 22:11:11.923794 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.923182    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="692ec1aa-ccc0-4712-88bd-b555b8eb9dae" containerName="istio-proxy"
Oct 29 22:11:11.923794 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.923377    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06268ed-ff58-4564-abd0-54609575a363" containerName="payload-pre-processing"
Oct 29 22:11:11.923794 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.923392    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58653d9-6bc5-44c4-97ed-068232825e36" containerName="istio-proxy"
Oct 29 22:11:11.923794 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.923407    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="b64e9d74-639d-44be-bdd1-04c9ef4f81fc" containerName="payload-pre-processing"
Oct 29 22:11:11.923794 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.923422    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="692ec1aa-ccc0-4712-88bd-b555b8eb9dae" containerName="istio-proxy"
Oct 29 22:11:11.923794 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.923436    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="f592a322-7cc5-41d0-8212-18797b23b843" containerName="payload-pre-processing"
Oct 29 22:11:11.923794 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.923449    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c4bfc57-2b32-410a-ab06-610e5749ab0e" containerName="payload-processing"
Oct 29 22:11:11.923794 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.923459    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="1693a089-535d-4f1d-815b-075639d9b9ab" containerName="istio-proxy"
Oct 29 22:11:11.923794 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.923469    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbcad86b-328d-49a0-a993-63dc68f154a4" containerName="main"
Oct 29 22:11:11.923794 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.923478    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ebf370-2322-49c9-8615-2d7109f57083" containerName="main"
Oct 29 22:11:11.927890 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.927871    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:11.947617 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.947573    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-4e8583-openshift-default-dockercfg-spxsd\""
Oct 29 22:11:11.949431 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.949393    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-workload-certs\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:11.949668 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.949647    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093f784b-0baf-4f14-984e-ec8003db30fb-istiod-ca-cert\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:11.949894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.949874    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-workload-socket\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:11.950145 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.950130    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093f784b-0baf-4f14-984e-ec8003db30fb-istio-podinfo\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:11.950359 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.950340    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-istio-data\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:11.950568 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.950549    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-credential-socket\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:11.950818 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.950800    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093f784b-0baf-4f14-984e-ec8003db30fb-istio-token\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:11.950978 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.950963    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvhsz\" (UniqueName: \"kubernetes.io/projected/093f784b-0baf-4f14-984e-ec8003db30fb-kube-api-access-kvhsz\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:11.951136 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.951122    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-istio-envoy\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:11.968553 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:11.968526    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"]
Oct 29 22:11:12.178314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:12.162662    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093f784b-0baf-4f14-984e-ec8003db30fb-istio-token\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:12.230162 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:12.230129    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvhsz\" (UniqueName: \"kubernetes.io/projected/093f784b-0baf-4f14-984e-ec8003db30fb-kube-api-access-kvhsz\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:12.230773 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:12.230742    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-istio-envoy\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:12.231107 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:12.231082    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-workload-certs\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:12.231311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:12.231269    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093f784b-0baf-4f14-984e-ec8003db30fb-istiod-ca-cert\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:12.231545 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:12.231529    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-workload-socket\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:12.231802 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:12.231787    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093f784b-0baf-4f14-984e-ec8003db30fb-istio-podinfo\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:12.231992 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:12.231972    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-istio-data\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:12.232195 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:12.232177    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-credential-socket\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:12.237459 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:12.232987    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-credential-socket\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:12.237613 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:12.233609    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-workload-socket\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:12.286654 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:12.237966    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-istio-data\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:12.286883 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:12.238177    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-workload-certs\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:12.287006 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:12.238913    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093f784b-0baf-4f14-984e-ec8003db30fb-istiod-ca-cert\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:12.287113 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:12.272992    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093f784b-0baf-4f14-984e-ec8003db30fb-istio-podinfo\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:12.287212 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:12.285242    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093f784b-0baf-4f14-984e-ec8003db30fb-istio-token\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:12.348385 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:12.334276    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvhsz\" (UniqueName: \"kubernetes.io/projected/093f784b-0baf-4f14-984e-ec8003db30fb-kube-api-access-kvhsz\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:12.429990 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:12.429871    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-istio-envoy\") pod \"e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") " pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:12.578741 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:12.578698    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:12.943904 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:11:12.943870    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093f784b_0baf_4f14_984e_ec8003db30fb.slice/crio-f1eb1382333bc3871f4e52025f6dc44d13a97a7ad06dc96743e54187adad64ab WatchSource:0}: Error finding container f1eb1382333bc3871f4e52025f6dc44d13a97a7ad06dc96743e54187adad64ab: Status 404 returned error can't find the container with id f1eb1382333bc3871f4e52025f6dc44d13a97a7ad06dc96743e54187adad64ab
Oct 29 22:11:12.944252 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:12.943932    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"]
Oct 29 22:11:12.946619 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:12.946587    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:11:12.946752 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:12.946664    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:11:12.946752 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:12.946710    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:11:13.215589 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:13.215482    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl" event={"ID":"093f784b-0baf-4f14-984e-ec8003db30fb","Type":"ContainerStarted","Data":"c6930380d3da248a77f1ea09cc7b648d3d2f5b54d53a107c727312115a29c31e"}
Oct 29 22:11:13.215589 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:13.215549    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl" event={"ID":"093f784b-0baf-4f14-984e-ec8003db30fb","Type":"ContainerStarted","Data":"f1eb1382333bc3871f4e52025f6dc44d13a97a7ad06dc96743e54187adad64ab"}
Oct 29 22:11:13.261661 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:13.261616    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl" podStartSLOduration=2.261602766 podStartE2EDuration="2.261602766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:13.259926468 +0000 UTC m=+1494.332883431" watchObservedRunningTime="2026-10-29 22:11:13.261602766 +0000 UTC m=+1494.334559729"
Oct 29 22:11:13.586524 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:13.580390    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:14.586350 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:14.586276    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:15.425717 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:15.424618    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:15.433310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:15.431790    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:11:15.873471 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:15.845370    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r"]
Oct 29 22:11:16.077666 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:16.076701    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r"
Oct 29 22:11:16.096716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:16.096678    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f82846\""
Oct 29 22:11:16.113911 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:16.113380    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f82846-dockercfg-gjjwb\""
Oct 29 22:11:16.145725 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:16.145626    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r"]
Oct 29 22:11:16.211028 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:16.210969    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2c8ef0-de16-4e8c-b524-21e8151dc910-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r\" (UID: \"4c2c8ef0-de16-4e8c-b524-21e8151dc910\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r"
Oct 29 22:11:16.211886 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:16.211854    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgfnx\" (UniqueName: \"kubernetes.io/projected/4c2c8ef0-de16-4e8c-b524-21e8151dc910-kube-api-access-cgfnx\") pod \"payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r\" (UID: \"4c2c8ef0-de16-4e8c-b524-21e8151dc910\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r"
Oct 29 22:11:16.314001 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:16.313960    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgfnx\" (UniqueName: \"kubernetes.io/projected/4c2c8ef0-de16-4e8c-b524-21e8151dc910-kube-api-access-cgfnx\") pod \"payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r\" (UID: \"4c2c8ef0-de16-4e8c-b524-21e8151dc910\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r"
Oct 29 22:11:16.314176 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:16.314102    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2c8ef0-de16-4e8c-b524-21e8151dc910-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r\" (UID: \"4c2c8ef0-de16-4e8c-b524-21e8151dc910\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r"
Oct 29 22:11:16.314864 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:16.314839    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2c8ef0-de16-4e8c-b524-21e8151dc910-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r\" (UID: \"4c2c8ef0-de16-4e8c-b524-21e8151dc910\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r"
Oct 29 22:11:16.336641 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:16.336617    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgfnx\" (UniqueName: \"kubernetes.io/projected/4c2c8ef0-de16-4e8c-b524-21e8151dc910-kube-api-access-cgfnx\") pod \"payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r\" (UID: \"4c2c8ef0-de16-4e8c-b524-21e8151dc910\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r"
Oct 29 22:11:16.416389 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:16.416368    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r"
Oct 29 22:11:16.791480 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:16.791451    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r"]
Oct 29 22:11:16.791623 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:11:16.791521    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c2c8ef0_de16_4e8c_b524_21e8151dc910.slice/crio-b942c084ee07b91ae53ca267d3039fb2ef839dae7594c5678c5e6c4dba9c9be1 WatchSource:0}: Error finding container b942c084ee07b91ae53ca267d3039fb2ef839dae7594c5678c5e6c4dba9c9be1: Status 404 returned error can't find the container with id b942c084ee07b91ae53ca267d3039fb2ef839dae7594c5678c5e6c4dba9c9be1
Oct 29 22:11:17.156626 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:17.156594    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4"]
Oct 29 22:11:17.189524 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:17.189497    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf"]
Oct 29 22:11:17.443033 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:17.442954    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r" event={"ID":"4c2c8ef0-de16-4e8c-b524-21e8151dc910","Type":"ContainerStarted","Data":"085bd13306ed47eac1f4efa4749cf1048de6f14c8db32d7d61872d7db75fbecf"}
Oct 29 22:11:17.443033 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:17.443015    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r" event={"ID":"4c2c8ef0-de16-4e8c-b524-21e8151dc910","Type":"ContainerStarted","Data":"b942c084ee07b91ae53ca267d3039fb2ef839dae7594c5678c5e6c4dba9c9be1"}
Oct 29 22:11:17.443236 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:17.443225    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r"
Oct 29 22:11:17.489168 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:17.489109    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r" podStartSLOduration=2.489091659 podStartE2EDuration="2.489091659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:17.482315677 +0000 UTC m=+1498.555272642" watchObservedRunningTime="2026-10-29 22:11:17.489091659 +0000 UTC m=+1498.562048623"
Oct 29 22:11:18.691850 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:18.691802    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj"]
Oct 29 22:11:18.697915 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:18.697890    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj"
Oct 29 22:11:18.718567 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:18.718534    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj"]
Oct 29 22:11:18.728006 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:18.727978    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-4e8583\""
Oct 29 22:11:18.775357 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:18.775326    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx"]
Oct 29 22:11:18.780251 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:18.780230    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx"
Oct 29 22:11:18.847325 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:18.836484    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-4e8583-dockercfg-qwvbx\""
Oct 29 22:11:18.992430 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:18.983226    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68dfa23b-de59-4864-a08e-7ab7185792a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx\" (UID: \"68dfa23b-de59-4864-a08e-7ab7185792a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx"
Oct 29 22:11:18.992430 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:18.983411    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0897e113-b10f-460b-9b0f-17ce9daea2c7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj\" (UID: \"0897e113-b10f-460b-9b0f-17ce9daea2c7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj"
Oct 29 22:11:18.992430 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:18.983647    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4szv9\" (UniqueName: \"kubernetes.io/projected/68dfa23b-de59-4864-a08e-7ab7185792a8-kube-api-access-4szv9\") pod \"payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx\" (UID: \"68dfa23b-de59-4864-a08e-7ab7185792a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx"
Oct 29 22:11:18.992430 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:18.983774    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rl5c\" (UniqueName: \"kubernetes.io/projected/0897e113-b10f-460b-9b0f-17ce9daea2c7-kube-api-access-7rl5c\") pod \"payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj\" (UID: \"0897e113-b10f-460b-9b0f-17ce9daea2c7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj"
Oct 29 22:11:19.106910 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:19.106873    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68dfa23b-de59-4864-a08e-7ab7185792a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx\" (UID: \"68dfa23b-de59-4864-a08e-7ab7185792a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx"
Oct 29 22:11:19.107317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:19.107275    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0897e113-b10f-460b-9b0f-17ce9daea2c7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj\" (UID: \"0897e113-b10f-460b-9b0f-17ce9daea2c7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj"
Oct 29 22:11:19.107798 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:19.107712    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4szv9\" (UniqueName: \"kubernetes.io/projected/68dfa23b-de59-4864-a08e-7ab7185792a8-kube-api-access-4szv9\") pod \"payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx\" (UID: \"68dfa23b-de59-4864-a08e-7ab7185792a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx"
Oct 29 22:11:19.195248 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:19.119525    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rl5c\" (UniqueName: \"kubernetes.io/projected/0897e113-b10f-460b-9b0f-17ce9daea2c7-kube-api-access-7rl5c\") pod \"payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj\" (UID: \"0897e113-b10f-460b-9b0f-17ce9daea2c7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj"
Oct 29 22:11:19.195248 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:19.120990    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0897e113-b10f-460b-9b0f-17ce9daea2c7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj\" (UID: \"0897e113-b10f-460b-9b0f-17ce9daea2c7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj"
Oct 29 22:11:19.195248 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:19.121969    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68dfa23b-de59-4864-a08e-7ab7185792a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx\" (UID: \"68dfa23b-de59-4864-a08e-7ab7185792a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx"
Oct 29 22:11:19.195248 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:19.192265    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4szv9\" (UniqueName: \"kubernetes.io/projected/68dfa23b-de59-4864-a08e-7ab7185792a8-kube-api-access-4szv9\") pod \"payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx\" (UID: \"68dfa23b-de59-4864-a08e-7ab7185792a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx"
Oct 29 22:11:19.209749 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:19.209682    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rl5c\" (UniqueName: \"kubernetes.io/projected/0897e113-b10f-460b-9b0f-17ce9daea2c7-kube-api-access-7rl5c\") pod \"payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj\" (UID: \"0897e113-b10f-460b-9b0f-17ce9daea2c7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj"
Oct 29 22:11:19.298035 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:19.297937    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx"]
Oct 29 22:11:19.314710 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:19.314678    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj"
Oct 29 22:11:19.425629 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:19.425592    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-4e8583-dockercfg-qwvbx\""
Oct 29 22:11:19.439678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:19.437659    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx"
Oct 29 22:11:19.667866 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:19.667835    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/2.log"
Oct 29 22:11:19.671186 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:19.671162    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/2.log"
Oct 29 22:11:19.678387 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:19.678369    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74bdq_259e6b95-d64c-4101-871f-4a933d4ac38f/ovn-acl-logging/0.log"
Oct 29 22:11:19.682102 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:19.682073    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74bdq_259e6b95-d64c-4101-871f-4a933d4ac38f/ovn-acl-logging/0.log"
Oct 29 22:11:19.782509 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:19.782480    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj"]
Oct 29 22:11:19.783148 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:11:19.783112    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0897e113_b10f_460b_9b0f_17ce9daea2c7.slice/crio-1a093c0f6cbde5f01ca4f6c06fc6ee29912409dad53d507a4e71b9e15d656257 WatchSource:0}: Error finding container 1a093c0f6cbde5f01ca4f6c06fc6ee29912409dad53d507a4e71b9e15d656257: Status 404 returned error can't find the container with id 1a093c0f6cbde5f01ca4f6c06fc6ee29912409dad53d507a4e71b9e15d656257
Oct 29 22:11:19.851628 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:19.851590    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx"]
Oct 29 22:11:19.852443 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:11:19.852414    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68dfa23b_de59_4864_a08e_7ab7185792a8.slice/crio-928a91dc7828cf0986927727d63784c761b4e64f9e0a80947abbd8506cee18b2 WatchSource:0}: Error finding container 928a91dc7828cf0986927727d63784c761b4e64f9e0a80947abbd8506cee18b2: Status 404 returned error can't find the container with id 928a91dc7828cf0986927727d63784c761b4e64f9e0a80947abbd8506cee18b2
Oct 29 22:11:20.667317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:20.648738    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj" event={"ID":"0897e113-b10f-460b-9b0f-17ce9daea2c7","Type":"ContainerStarted","Data":"59aa8976d5f4d879aa9f6c90c864bd8c434090e97702b617a871d7814063a3af"}
Oct 29 22:11:20.667731 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:20.667687    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj" event={"ID":"0897e113-b10f-460b-9b0f-17ce9daea2c7","Type":"ContainerStarted","Data":"1a093c0f6cbde5f01ca4f6c06fc6ee29912409dad53d507a4e71b9e15d656257"}
Oct 29 22:11:20.669564 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:20.669542    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj"
Oct 29 22:11:20.689268 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:20.689223    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx" event={"ID":"68dfa23b-de59-4864-a08e-7ab7185792a8","Type":"ContainerStarted","Data":"4419e1615d1af83169c7ff79d7d55a8f443b7720f986d8de658ffa33f69e3c7a"}
Oct 29 22:11:20.691212 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:20.691185    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx" event={"ID":"68dfa23b-de59-4864-a08e-7ab7185792a8","Type":"ContainerStarted","Data":"928a91dc7828cf0986927727d63784c761b4e64f9e0a80947abbd8506cee18b2"}
Oct 29 22:11:20.691668 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:20.691610    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx"
Oct 29 22:11:20.881600 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:20.881538    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj" podStartSLOduration=2.881516688 podStartE2EDuration="2.881516688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:20.725884501 +0000 UTC m=+1501.798841466" watchObservedRunningTime="2026-10-29 22:11:20.881516688 +0000 UTC m=+1501.954473655"
Oct 29 22:11:20.957875 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:20.957759    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx" podStartSLOduration=2.957736432 podStartE2EDuration="2.957736432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:20.783249603 +0000 UTC m=+1501.856206568" watchObservedRunningTime="2026-10-29 22:11:20.957736432 +0000 UTC m=+1502.030693397"
Oct 29 22:11:22.156909 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.156860    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4" podUID="81379dd3-6066-452f-868b-8bde1172bd56" containerName="payload-processing" containerID="cri-o://c1d1488819b8ceccb0b0a3e43831aabc3b681bac78198ff880f12901365eb62f" gracePeriod=25
Oct 29 22:11:22.190804 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.190771    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf" podUID="50625ced-c86d-42a4-ab2d-7cff60f95ce6" containerName="payload-pre-processing" containerID="cri-o://a75a1cfbf825b281245b6fce21c42f605fca5811d5a644ffd276681440cbfbf1" gracePeriod=25
Oct 29 22:11:22.485168 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.485139    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4"
Oct 29 22:11:22.521029 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.521007    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf"
Oct 29 22:11:22.590440 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.590409    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50625ced-c86d-42a4-ab2d-7cff60f95ce6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50625ced-c86d-42a4-ab2d-7cff60f95ce6-plugins-config-volume\") pod \"50625ced-c86d-42a4-ab2d-7cff60f95ce6\" (UID: \"50625ced-c86d-42a4-ab2d-7cff60f95ce6\") "
Oct 29 22:11:22.590641 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.590548    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81379dd3-6066-452f-868b-8bde1172bd56-kube-api-access-tj564\" (UniqueName: \"kubernetes.io/projected/81379dd3-6066-452f-868b-8bde1172bd56-kube-api-access-tj564\") pod \"81379dd3-6066-452f-868b-8bde1172bd56\" (UID: \"81379dd3-6066-452f-868b-8bde1172bd56\") "
Oct 29 22:11:22.590818 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.590790    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81379dd3-6066-452f-868b-8bde1172bd56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81379dd3-6066-452f-868b-8bde1172bd56-plugins-config-volume\") pod \"81379dd3-6066-452f-868b-8bde1172bd56\" (UID: \"81379dd3-6066-452f-868b-8bde1172bd56\") "
Oct 29 22:11:22.590928 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.590911    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50625ced-c86d-42a4-ab2d-7cff60f95ce6-kube-api-access-rflwb\" (UniqueName: \"kubernetes.io/projected/50625ced-c86d-42a4-ab2d-7cff60f95ce6-kube-api-access-rflwb\") pod \"50625ced-c86d-42a4-ab2d-7cff60f95ce6\" (UID: \"50625ced-c86d-42a4-ab2d-7cff60f95ce6\") "
Oct 29 22:11:22.591403 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.591347    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50625ced-c86d-42a4-ab2d-7cff60f95ce6-plugins-config-volume" pod "50625ced-c86d-42a4-ab2d-7cff60f95ce6" (UID: "50625ced-c86d-42a4-ab2d-7cff60f95ce6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:22.591403 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.591379    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81379dd3-6066-452f-868b-8bde1172bd56-plugins-config-volume" pod "81379dd3-6066-452f-868b-8bde1172bd56" (UID: "81379dd3-6066-452f-868b-8bde1172bd56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:22.592021 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.591991    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50625ced-c86d-42a4-ab2d-7cff60f95ce6-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:22.592126 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.592034    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81379dd3-6066-452f-868b-8bde1172bd56-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:22.593565 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.593531    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81379dd3-6066-452f-868b-8bde1172bd56-kube-api-access-tj564" pod "81379dd3-6066-452f-868b-8bde1172bd56" (UID: "81379dd3-6066-452f-868b-8bde1172bd56"). InnerVolumeSpecName "kube-api-access-tj564". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:22.593675 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.593636    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50625ced-c86d-42a4-ab2d-7cff60f95ce6-kube-api-access-rflwb" pod "50625ced-c86d-42a4-ab2d-7cff60f95ce6" (UID: "50625ced-c86d-42a4-ab2d-7cff60f95ce6"). InnerVolumeSpecName "kube-api-access-rflwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:22.702373 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.701524    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rflwb\" (UniqueName: \"kubernetes.io/projected/50625ced-c86d-42a4-ab2d-7cff60f95ce6-kube-api-access-rflwb\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:22.702373 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.701568    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj564\" (UniqueName: \"kubernetes.io/projected/81379dd3-6066-452f-868b-8bde1172bd56-kube-api-access-tj564\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:22.793214 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.793173    4962 generic.go:356] "Generic (PLEG): container finished" podID="81379dd3-6066-452f-868b-8bde1172bd56" containerID="c1d1488819b8ceccb0b0a3e43831aabc3b681bac78198ff880f12901365eb62f" exitCode=0
Oct 29 22:11:22.793561 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.793528    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4" event={"ID":"81379dd3-6066-452f-868b-8bde1172bd56","Type":"ContainerDied","Data":"c1d1488819b8ceccb0b0a3e43831aabc3b681bac78198ff880f12901365eb62f"}
Oct 29 22:11:22.793736 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.793720    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4" event={"ID":"81379dd3-6066-452f-868b-8bde1172bd56","Type":"ContainerDied","Data":"ee1f51baa788fb14a8c91453e1a8a07d5f10e10f249941965e57dfe793762124"}
Oct 29 22:11:22.793847 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.793835    4962 scope.go:122] "RemoveContainer" containerID="c1d1488819b8ceccb0b0a3e43831aabc3b681bac78198ff880f12901365eb62f"
Oct 29 22:11:22.794120 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.794102    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4"
Oct 29 22:11:22.813097 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.809176    4962 generic.go:356] "Generic (PLEG): container finished" podID="50625ced-c86d-42a4-ab2d-7cff60f95ce6" containerID="a75a1cfbf825b281245b6fce21c42f605fca5811d5a644ffd276681440cbfbf1" exitCode=0
Oct 29 22:11:22.813097 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.810402    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf"
Oct 29 22:11:22.813097 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.810516    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf" event={"ID":"50625ced-c86d-42a4-ab2d-7cff60f95ce6","Type":"ContainerDied","Data":"a75a1cfbf825b281245b6fce21c42f605fca5811d5a644ffd276681440cbfbf1"}
Oct 29 22:11:22.813097 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.810571    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf" event={"ID":"50625ced-c86d-42a4-ab2d-7cff60f95ce6","Type":"ContainerDied","Data":"cf2cfef6d61fe0838449cbb3aae02622372a87d413b14d79a6f067d9a889da82"}
Oct 29 22:11:22.826203 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.826181    4962 scope.go:122] "RemoveContainer" containerID="c1d1488819b8ceccb0b0a3e43831aabc3b681bac78198ff880f12901365eb62f"
Oct 29 22:11:22.826554 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:11:22.826521    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1d1488819b8ceccb0b0a3e43831aabc3b681bac78198ff880f12901365eb62f\": container with ID starting with c1d1488819b8ceccb0b0a3e43831aabc3b681bac78198ff880f12901365eb62f not found: ID does not exist" containerID="c1d1488819b8ceccb0b0a3e43831aabc3b681bac78198ff880f12901365eb62f"
Oct 29 22:11:22.826677 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.826564    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1d1488819b8ceccb0b0a3e43831aabc3b681bac78198ff880f12901365eb62f"} err="failed to get container status \"c1d1488819b8ceccb0b0a3e43831aabc3b681bac78198ff880f12901365eb62f\": rpc error: code = NotFound desc = could not find container \"c1d1488819b8ceccb0b0a3e43831aabc3b681bac78198ff880f12901365eb62f\": container with ID starting with c1d1488819b8ceccb0b0a3e43831aabc3b681bac78198ff880f12901365eb62f not found: ID does not exist"
Oct 29 22:11:22.826677 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.826588    4962 scope.go:122] "RemoveContainer" containerID="a75a1cfbf825b281245b6fce21c42f605fca5811d5a644ffd276681440cbfbf1"
Oct 29 22:11:22.843277 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.843258    4962 scope.go:122] "RemoveContainer" containerID="a75a1cfbf825b281245b6fce21c42f605fca5811d5a644ffd276681440cbfbf1"
Oct 29 22:11:22.843653 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:11:22.843628    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a75a1cfbf825b281245b6fce21c42f605fca5811d5a644ffd276681440cbfbf1\": container with ID starting with a75a1cfbf825b281245b6fce21c42f605fca5811d5a644ffd276681440cbfbf1 not found: ID does not exist" containerID="a75a1cfbf825b281245b6fce21c42f605fca5811d5a644ffd276681440cbfbf1"
Oct 29 22:11:22.843721 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.843663    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a75a1cfbf825b281245b6fce21c42f605fca5811d5a644ffd276681440cbfbf1"} err="failed to get container status \"a75a1cfbf825b281245b6fce21c42f605fca5811d5a644ffd276681440cbfbf1\": rpc error: code = NotFound desc = could not find container \"a75a1cfbf825b281245b6fce21c42f605fca5811d5a644ffd276681440cbfbf1\": container with ID starting with a75a1cfbf825b281245b6fce21c42f605fca5811d5a644ffd276681440cbfbf1 not found: ID does not exist"
Oct 29 22:11:22.872765 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.872737    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4"]
Oct 29 22:11:22.881200 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.881171    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9ae129-b8dc59c64-gtpj4"]
Oct 29 22:11:22.911797 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.911769    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf"]
Oct 29 22:11:22.917517 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:22.917494    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9ae129-d5cbb7647-qnnmf"]
Oct 29 22:11:23.061481 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:23.061096    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50625ced-c86d-42a4-ab2d-7cff60f95ce6" path="/var/lib/kubelet/pods/50625ced-c86d-42a4-ab2d-7cff60f95ce6/volumes"
Oct 29 22:11:23.083439 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:23.082811    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81379dd3-6066-452f-868b-8bde1172bd56" path="/var/lib/kubelet/pods/81379dd3-6066-452f-868b-8bde1172bd56/volumes"
Oct 29 22:11:27.458359 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:27.458326    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf"]
Oct 29 22:11:27.458851 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:27.458527    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf" podUID="7cb302c3-d428-4a64-a44e-d5661108233d" containerName="maas-api" containerID="cri-o://5834de82e79b26260dcc022d78a1e49b362baa587ea17e68f6b5fbb609230257" gracePeriod=30
Oct 29 22:11:28.354528 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:28.354320    4962 generic.go:356] "Generic (PLEG): container finished" podID="7cb302c3-d428-4a64-a44e-d5661108233d" containerID="5834de82e79b26260dcc022d78a1e49b362baa587ea17e68f6b5fbb609230257" exitCode=0
Oct 29 22:11:28.354528 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:28.354435    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf" event={"ID":"7cb302c3-d428-4a64-a44e-d5661108233d","Type":"ContainerDied","Data":"5834de82e79b26260dcc022d78a1e49b362baa587ea17e68f6b5fbb609230257"}
Oct 29 22:11:28.397071 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:28.387012    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf"
Oct 29 22:11:28.531143 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:28.486002    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cb302c3-d428-4a64-a44e-d5661108233d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cb302c3-d428-4a64-a44e-d5661108233d-metrics-certs\") pod \"7cb302c3-d428-4a64-a44e-d5661108233d\" (UID: \"7cb302c3-d428-4a64-a44e-d5661108233d\") "
Oct 29 22:11:28.531143 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:28.486073    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb302c3-d428-4a64-a44e-d5661108233d-kube-api-access-5cx95\" (UniqueName: \"kubernetes.io/projected/7cb302c3-d428-4a64-a44e-d5661108233d-kube-api-access-5cx95\") pod \"7cb302c3-d428-4a64-a44e-d5661108233d\" (UID: \"7cb302c3-d428-4a64-a44e-d5661108233d\") "
Oct 29 22:11:28.531143 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:28.486218    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cb302c3-d428-4a64-a44e-d5661108233d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb302c3-d428-4a64-a44e-d5661108233d-openshift-service-ca\") pod \"7cb302c3-d428-4a64-a44e-d5661108233d\" (UID: \"7cb302c3-d428-4a64-a44e-d5661108233d\") "
Oct 29 22:11:28.531143 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:28.486475    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cb302c3-d428-4a64-a44e-d5661108233d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cb302c3-d428-4a64-a44e-d5661108233d-maas-api-tls\") pod \"7cb302c3-d428-4a64-a44e-d5661108233d\" (UID: \"7cb302c3-d428-4a64-a44e-d5661108233d\") "
Oct 29 22:11:28.531143 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:28.499025    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r"
Oct 29 22:11:28.531143 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:28.502499    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb302c3-d428-4a64-a44e-d5661108233d-openshift-service-ca" pod "7cb302c3-d428-4a64-a44e-d5661108233d" (UID: "7cb302c3-d428-4a64-a44e-d5661108233d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:28.531143 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:28.505807    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cb302c3-d428-4a64-a44e-d5661108233d-maas-api-tls" pod "7cb302c3-d428-4a64-a44e-d5661108233d" (UID: "7cb302c3-d428-4a64-a44e-d5661108233d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:11:28.538871 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:28.537388    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb302c3-d428-4a64-a44e-d5661108233d-kube-api-access-5cx95" pod "7cb302c3-d428-4a64-a44e-d5661108233d" (UID: "7cb302c3-d428-4a64-a44e-d5661108233d"). InnerVolumeSpecName "kube-api-access-5cx95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:28.543481 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:28.541619    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cb302c3-d428-4a64-a44e-d5661108233d-metrics-certs" pod "7cb302c3-d428-4a64-a44e-d5661108233d" (UID: "7cb302c3-d428-4a64-a44e-d5661108233d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:11:28.589787 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:28.588725    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cb302c3-d428-4a64-a44e-d5661108233d-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:28.589787 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:28.588759    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cb302c3-d428-4a64-a44e-d5661108233d-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:28.589787 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:28.588776    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cx95\" (UniqueName: \"kubernetes.io/projected/7cb302c3-d428-4a64-a44e-d5661108233d-kube-api-access-5cx95\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:28.589787 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:28.588792    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cb302c3-d428-4a64-a44e-d5661108233d-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:29.150946 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.150916    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"]
Oct 29 22:11:29.151816 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.151791    4962 cpu_manager.go:414] "containerMap: removing container" podUID="81379dd3-6066-452f-868b-8bde1172bd56" containerName="payload-processing"
Oct 29 22:11:29.151816 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.151815    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81379dd3-6066-452f-868b-8bde1172bd56" containerName="payload-processing"
Oct 29 22:11:29.152037 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.151836    4962 cpu_manager.go:414] "containerMap: removing container" podUID="7cb302c3-d428-4a64-a44e-d5661108233d" containerName="maas-api"
Oct 29 22:11:29.152037 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.151841    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb302c3-d428-4a64-a44e-d5661108233d" containerName="maas-api"
Oct 29 22:11:29.152037 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.151860    4962 cpu_manager.go:414] "containerMap: removing container" podUID="50625ced-c86d-42a4-ab2d-7cff60f95ce6" containerName="payload-pre-processing"
Oct 29 22:11:29.152037 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.151865    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50625ced-c86d-42a4-ab2d-7cff60f95ce6" containerName="payload-pre-processing"
Oct 29 22:11:29.152251 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.152044    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="81379dd3-6066-452f-868b-8bde1172bd56" containerName="payload-processing"
Oct 29 22:11:29.152251 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.152063    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb302c3-d428-4a64-a44e-d5661108233d" containerName="maas-api"
Oct 29 22:11:29.152251 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.152080    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="50625ced-c86d-42a4-ab2d-7cff60f95ce6" containerName="payload-pre-processing"
Oct 29 22:11:29.156180 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.156162    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.163971 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.163933    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"]
Oct 29 22:11:29.168909 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.168890    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.184062 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.184040    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"]
Oct 29 22:11:29.194422 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.194398    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"]
Oct 29 22:11:29.240140 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.240101    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-podinfo\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.240340 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.240235    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-podinfo\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.240395 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.240347    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bxt4\" (UniqueName: \"kubernetes.io/projected/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-kube-api-access-2bxt4\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.240431 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.240395    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-workload-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.240482 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.240445    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-envoy\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.240540 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.240500    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c584046d-f7a2-4cda-9965-e95d0dfe944d-istiod-ca-cert\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.240584 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.240540    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-workload-certs\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.240659 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.240643    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-token\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.240749 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.240735    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-credential-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.240793 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.240759    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-data\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.240826 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.240812    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istiod-ca-cert\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.240867 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.240860    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-credential-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.240910 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.240883    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-666vn\" (UniqueName: \"kubernetes.io/projected/c584046d-f7a2-4cda-9965-e95d0dfe944d-kube-api-access-666vn\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.240950 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.240943    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-envoy\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.241015 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.240967    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-workload-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.241015 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.240992    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-workload-certs\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.241015 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.241007    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-data\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.241116 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.241034    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-token\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.342961 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.342893    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c584046d-f7a2-4cda-9965-e95d0dfe944d-istiod-ca-cert\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.342961 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.342930    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-workload-certs\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.343174 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.342965    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-token\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.343174 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.343053    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-credential-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.343174 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.343075    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-data\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.343174 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.343129    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istiod-ca-cert\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.343449 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.343198    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-credential-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.343449 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.343226    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-666vn\" (UniqueName: \"kubernetes.io/projected/c584046d-f7a2-4cda-9965-e95d0dfe944d-kube-api-access-666vn\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.343449 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.343347    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-envoy\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.343449 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.343384    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-workload-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.343449 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.343422    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-workload-certs\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.343449 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.343436    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-workload-certs\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.343779 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.343481    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-credential-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.343779 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.343445    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-data\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.343779 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.343501    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-data\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.343779 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.343529    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-credential-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.343779 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.343545    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-token\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.343779 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.343600    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-podinfo\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.343779 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.343672    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-podinfo\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.343779 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.343703    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bxt4\" (UniqueName: \"kubernetes.io/projected/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-kube-api-access-2bxt4\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.343779 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.343703    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c584046d-f7a2-4cda-9965-e95d0dfe944d-istiod-ca-cert\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.343779 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.343726    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-data\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.343779 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.343770    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-workload-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.343779 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.343776    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-workload-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.344612 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.343802    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-workload-certs\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.344612 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.343932    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-envoy\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.344612 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.343995    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istiod-ca-cert\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.344612 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.344079    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-workload-socket\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.345879 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.345855    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-envoy\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.346236 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.346210    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-envoy\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.346372 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.346354    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-podinfo\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.346435 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.346386    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-podinfo\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.367011 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.366987    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf"
Oct 29 22:11:29.367119 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.367008    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf" event={"ID":"7cb302c3-d428-4a64-a44e-d5661108233d","Type":"ContainerDied","Data":"c1600b8900822044a95043d99ce407d1bbabcf2acbbc91dd751eead5022ec582"}
Oct 29 22:11:29.367119 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.367072    4962 scope.go:122] "RemoveContainer" containerID="5834de82e79b26260dcc022d78a1e49b362baa587ea17e68f6b5fbb609230257"
Oct 29 22:11:29.368106 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.368082    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-token\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.368106 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.368090    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-token\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.368382 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.368357    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-666vn\" (UniqueName: \"kubernetes.io/projected/c584046d-f7a2-4cda-9965-e95d0dfe944d-kube-api-access-666vn\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.368475 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.368383    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bxt4\" (UniqueName: \"kubernetes.io/projected/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-kube-api-access-2bxt4\") pod \"e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") " pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.433599 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.433573    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf"]
Oct 29 22:11:29.437746 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.437725    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a3c02e-6f8f9867c4-5fqlf"]
Oct 29 22:11:29.474043 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.474019    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:29.486921 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.486901    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:29.938433 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:29.935791    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"]
Oct 29 22:11:30.016050 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:11:30.015542    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc584046d_f7a2_4cda_9965_e95d0dfe944d.slice/crio-07d8326d905e4fa4cc4ff0a6945d9f78bec2e0a269fdc6018ca995af53979bf1 WatchSource:0}: Error finding container 07d8326d905e4fa4cc4ff0a6945d9f78bec2e0a269fdc6018ca995af53979bf1: Status 404 returned error can't find the container with id 07d8326d905e4fa4cc4ff0a6945d9f78bec2e0a269fdc6018ca995af53979bf1
Oct 29 22:11:30.018447 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:30.018411    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"]
Oct 29 22:11:30.088329 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:30.086363    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:11:30.088329 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:30.086784    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:11:30.088329 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:30.087580    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:11:30.120967 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:30.119910    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:11:30.120967 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:30.120049    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:11:30.120967 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:30.120157    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:11:30.380363 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:30.378913    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9" event={"ID":"04500ffa-4da1-4d50-8fee-4e00ad03f4b0","Type":"ContainerStarted","Data":"8d4f497475dd9cbd8413f35b5f3f90cf13bdf6084173a78a31d0c07b31d7ff30"}
Oct 29 22:11:30.380363 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:30.379054    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9" event={"ID":"04500ffa-4da1-4d50-8fee-4e00ad03f4b0","Type":"ContainerStarted","Data":"80e1b63bd97e70f8b321f1a094323d5024068f8094ad6851a9b03b66b5fdb2a8"}
Oct 29 22:11:30.382220 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:30.382190    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" event={"ID":"c584046d-f7a2-4cda-9965-e95d0dfe944d","Type":"ContainerStarted","Data":"81520d8cb39c5982bf8fb6f764297eabd7da42c56858f6fadfc97a5fc45cde0b"}
Oct 29 22:11:30.382422 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:30.382238    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" event={"ID":"c584046d-f7a2-4cda-9965-e95d0dfe944d","Type":"ContainerStarted","Data":"07d8326d905e4fa4cc4ff0a6945d9f78bec2e0a269fdc6018ca995af53979bf1"}
Oct 29 22:11:30.485367 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:30.479803    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:30.517311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:30.499789    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:30.517311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:30.502583    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9" podStartSLOduration=1.502561752 podStartE2EDuration="1.502561752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:30.430795377 +0000 UTC m=+1511.503752342" watchObservedRunningTime="2026-10-29 22:11:30.502561752 +0000 UTC m=+1511.575518709"
Oct 29 22:11:30.525310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:30.519266    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" podStartSLOduration=1.519245586 podStartE2EDuration="1.519245586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:30.483082868 +0000 UTC m=+1511.556039833" watchObservedRunningTime="2026-10-29 22:11:30.519245586 +0000 UTC m=+1511.592202548"
Oct 29 22:11:31.060947 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:31.060911    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cb302c3-d428-4a64-a44e-d5661108233d" path="/var/lib/kubelet/pods/7cb302c3-d428-4a64-a44e-d5661108233d/volumes"
Oct 29 22:11:31.502323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:31.501932    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:11:31.502323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:31.501994    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9" podUID="04500ffa-4da1-4d50-8fee-4e00ad03f4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 29 22:11:31.505317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:31.502995    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:11:31.505317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:31.503042    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:11:32.495759 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:32.488992    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:11:32.495759 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:32.489067    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:11:32.495759 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:32.491182    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:11:32.495759 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:32.491224    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9" podUID="04500ffa-4da1-4d50-8fee-4e00ad03f4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 29 22:11:32.822954 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:32.821453    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj"
Oct 29 22:11:32.864529 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:32.864494    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx"
Oct 29 22:11:33.530170 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:33.529418    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:33.599020 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:33.598319    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:11:33.599020 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:33.598380    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9" podUID="04500ffa-4da1-4d50-8fee-4e00ad03f4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 29 22:11:33.599020 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:33.598773    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:11:33.599020 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:33.598813    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:11:34.200335 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:34.197394    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:11:34.254310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:34.245618    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:34.355201 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:34.254454    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:11:34.355201 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:34.308504    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded"
Oct 29 22:11:34.443636 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:34.443587    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": context deadline exceeded"
Oct 29 22:11:34.559355 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:34.559224    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:34.587185 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:34.587135    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:11:34.587566 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:34.587519    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:11:34.633173 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:34.633028    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:11:34.633532 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:34.633472    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9" podUID="04500ffa-4da1-4d50-8fee-4e00ad03f4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 29 22:11:34.646313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:34.641021    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:11:34.879739 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:11:34.879650    4962 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.66s"
Oct 29 22:11:34.880039 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:34.880023    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 22:11:34.880801 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:34.880779    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"]
Oct 29 22:11:34.928460 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:34.928407    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:34.971340 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:34.971282    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.039064 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.038618    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"]
Oct 29 22:11:35.132922 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.132809    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:11:35.163454 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.163393    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:35.206662 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.206625    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.334313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.218247    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cda533-309f-46e1-8cad-7239630ac19d-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.360469 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.360433    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.360945 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.360919    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.386361 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.361180    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.386737 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.386707    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfrhg\" (UniqueName: \"kubernetes.io/projected/b4cda533-309f-46e1-8cad-7239630ac19d-kube-api-access-nfrhg\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.386959 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.386939    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cda533-309f-46e1-8cad-7239630ac19d-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.387136 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.387119    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cda533-309f-46e1-8cad-7239630ac19d-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.387422 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.387398    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.387893 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.360321    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:35.388554 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.372658    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:35.398989 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.332670    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:35.399431 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.388003    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.423710 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.423406    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:35.594937 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.593464    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:11:35.594937 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.593527    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9" podUID="04500ffa-4da1-4d50-8fee-4e00ad03f4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 29 22:11:35.594937 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.593983    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:35.641716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.613843    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfrhg\" (UniqueName: \"kubernetes.io/projected/b4cda533-309f-46e1-8cad-7239630ac19d-kube-api-access-nfrhg\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.641716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.639011    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cda533-309f-46e1-8cad-7239630ac19d-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.641716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.639068    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cda533-309f-46e1-8cad-7239630ac19d-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.641716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.639134    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j8bh\" (UniqueName: \"kubernetes.io/projected/e8c4fd64-8818-475c-b3d0-579054bcd76a-kube-api-access-6j8bh\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.641716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.639186    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.641716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.639210    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.641716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.639238    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.641716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.639329    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.641716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.639411    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.641716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.639445    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c4fd64-8818-475c-b3d0-579054bcd76a-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.641716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.639473    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.641716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.639656    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cda533-309f-46e1-8cad-7239630ac19d-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.641716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.639684    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.641716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.639711    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.641716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.639761    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.641716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.639844    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.641716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.639925    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.641716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.639977    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.641716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.641228    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cda533-309f-46e1-8cad-7239630ac19d-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.670617 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.670493    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:35.689815 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.689782    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:11:35.734313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.724387    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:11:35.776113 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.705198    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.892074 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.717300    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.892303 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.717547    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.892425 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.735444    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cda533-309f-46e1-8cad-7239630ac19d-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.892523 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.704828    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfrhg\" (UniqueName: \"kubernetes.io/projected/b4cda533-309f-46e1-8cad-7239630ac19d-kube-api-access-nfrhg\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.892636 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.748757    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.892728 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.750594    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cda533-309f-46e1-8cad-7239630ac19d-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.892970 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.750968    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.893092 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.810967    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.893251 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.893230    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.893566 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.893543    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.893894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.893871    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j8bh\" (UniqueName: \"kubernetes.io/projected/e8c4fd64-8818-475c-b3d0-579054bcd76a-kube-api-access-6j8bh\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.894042 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.894025    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.894167 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.894149    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.894310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.894275    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.916379 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.916351    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.916935 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.916912    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c4fd64-8818-475c-b3d0-579054bcd76a-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.940336 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.913762    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-workload-certs\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.940569 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.914059    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-credential-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.940708 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.868998    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"]
Oct 29 22:11:35.940927 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.940910    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:11:35.941436 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.941417    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 22:11:35.941610 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.941596    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:11:35.941752 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.941740    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:11:35.941865 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.941853    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"]
Oct 29 22:11:35.942981 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.905708    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-workload-socket\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.966430 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.933382    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:35.967720 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.967463    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-podinfo\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.967720 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.916840    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-data\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.967937 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.939840    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c4fd64-8818-475c-b3d0-579054bcd76a-istiod-ca-cert\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:35.968260 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.968240    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:11:35.990555 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.972524    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:35.996310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:35.994176    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-token\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:36.006309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:36.006263    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j8bh\" (UniqueName: \"kubernetes.io/projected/e8c4fd64-8818-475c-b3d0-579054bcd76a-kube-api-access-6j8bh\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:36.047311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:36.044756    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-envoy\") pod \"e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") " pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:36.154548 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:36.154491    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:36.164543 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:36.164505    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 22:11:36.167647 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:36.167614    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:36.183312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:36.168610    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:36.192330 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:36.169843    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 22:11:36.388857 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:36.388815    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:11:36.389752 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:36.389711    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:11:36.400428 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:36.400060    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:36.428831 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:36.425349    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:36.428831 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:36.426012    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:11:36.478722 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:36.478685    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:11:36.478962 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:36.478935    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9" podUID="04500ffa-4da1-4d50-8fee-4e00ad03f4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 29 22:11:36.504450 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:36.503575    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:11:36.504450 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:36.503639    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:11:36.595405 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:36.595363    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:36.687532 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:36.687424    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:36.836699 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:36.836346    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/health\": context deadline exceeded"
Oct 29 22:11:37.003383 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:36.977390    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:11:37.156238 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:37.155968    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:37.241312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:37.185428    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:11:37.476555 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:37.423858    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:11:37.542403 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:37.516127    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": context deadline exceeded"
Oct 29 22:11:37.623356 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:37.574784    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:11:37.623356 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:37.576587    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:11:37.651602 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:37.650960    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:11:37.651602 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:37.651017    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:11:37.750960 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:37.653237    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:11:37.803317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:37.687407    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:11:37.803317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:37.790850    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9" podUID="04500ffa-4da1-4d50-8fee-4e00ad03f4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 29 22:11:37.803317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:37.688101    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:37.886736 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:37.886678    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded"
Oct 29 22:11:38.006618 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:38.006516    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:38.210979 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:38.196641    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:38.348421 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:11:38.299830    4962 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.217s"
Oct 29 22:11:38.438315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:38.385197    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:11:38.475745 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:38.475712    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:11:38.476051 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:38.476004    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9" podUID="04500ffa-4da1-4d50-8fee-4e00ad03f4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 29 22:11:38.491653 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:38.491614    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:11:38.491973 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:38.491924    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:11:38.531118 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:38.531080    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:38.532335 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:38.531380    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:11:38.607596 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:38.607500    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:11:38.611395 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:38.611353    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:11:38.867412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:38.859431    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:11:38.917634 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:38.867980    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:39.066350 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:39.017118    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:39.185435 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:39.124423    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:39.226469 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:39.218392    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:11:39.487857 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:39.487755    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:11:39.648139 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:39.553566    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:11:39.908366 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:39.866410    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9" podUID="04500ffa-4da1-4d50-8fee-4e00ad03f4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 29 22:11:39.986349 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:39.773016    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:11:39.990716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:39.865329    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:40.018149 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:39.890727    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:40.109112 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:39.610638    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:11:40.129775 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:39.679637    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:11:40.130092 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:40.130043    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:11:40.354476 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:39.554274    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:40.361226 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:40.361165    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:11:40.532314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:40.493882    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:11:40.532314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:40.526215    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr" podUID="4ef87fe2-429a-4393-8ddf-fa43e2dd9044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:40.532314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:40.529373    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:11:40.576314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:40.568888    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:11:40.576314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:40.568971    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9" podUID="04500ffa-4da1-4d50-8fee-4e00ad03f4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 29 22:11:40.702042 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:40.688245    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:11:40.787127 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:40.747107    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:11:40.791225 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:11:40.790476    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:11:40.869470 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:40.831798    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:11:40.944311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:40.937795    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:11:40.945986 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:11:40.945956    4962 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.834s"
Oct 29 22:11:40.946162 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:40.946146    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"]
Oct 29 22:11:40.946342 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:40.946327    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"]
Oct 29 22:11:40.946671 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:40.946655    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"]
Oct 29 22:11:40.950306 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:40.947021    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc" podUID="35c2b463-e2a6-4b1d-8425-558ca7832668" containerName="istio-proxy" containerID="cri-o://ee42b17744a6a1fc61efde1f2211e522f8707a0f8a5b4888967a0acae6e3ab33" gracePeriod=30
Oct 29 22:11:40.959618 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:40.950755    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42" podUID="e1897161-0016-400a-9928-8b407f4148e9" containerName="istio-proxy" containerID="cri-o://2fb3d9576ee00f91c232b0db8a4d81c05a44ba966996f0907fb75967ce88eb72" gracePeriod=30
Oct 29 22:11:41.077692 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:41.046522    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:41.084016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:41.083970    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc" podUID="036f71f6-61b8-469d-a080-d1ab0b45c195" containerName="istio-proxy" containerID="cri-o://271b7b89784cda842fead66ac67418b72e6abd7145340569ffe392ef431c8b62" gracePeriod=30
Oct 29 22:11:41.203540 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:41.139454    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:11:41.203540 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:41.140418    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:11:41.317751 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:40.993961    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:41.373348 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:41.360368    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:41.538326 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:41.532564    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:41.538326 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:41.534011    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:41.751017 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:41.750753    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:11:41.751017 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:41.750932    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded"
Oct 29 22:11:42.163312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:42.157538    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg" podUID="70e1bafe-4b93-44d7-a179-44d5a82fd069" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:42.164949 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:42.164905    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:42.165545 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:42.165519    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded"
Oct 29 22:11:42.165798 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:42.165775    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": context deadline exceeded"
Oct 29 22:11:42.359716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:42.337495    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:42.359716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:42.337889    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:11:42.359716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:42.337954    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw" podUID="f6631eff-5d71-483c-aac7-8d578274ce76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:42.380471 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:42.380398    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:42.439314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:42.421573    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:42.673681 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:42.673646    4962 trace.go:236] Trace[671680960]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc" (29-Oct-2026 22:11:41.442) (total time: 1231ms):
Oct 29 22:11:42.673681 ip-10-0-129-146 kubenswrapper[4962]: Trace[671680960]: [1.23104548s] [1.23104548s] END
Oct 29 22:11:42.712740 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:42.712692    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:11:42.713395 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:42.713361    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:11:42.713816 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:42.713791    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:11:42.713946 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:42.713927    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9" podUID="04500ffa-4da1-4d50-8fee-4e00ad03f4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:42.834336 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:42.797357    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:11:42.834336 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:42.797718    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:11:42.834336 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:42.797759    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:42.834336 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:42.798006    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:11:42.844355 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:42.837345    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:11:42.844355 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:42.837398    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:11:42.844355 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:42.839273    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:11:42.844355 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:42.839332    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9" podUID="04500ffa-4da1-4d50-8fee-4e00ad03f4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 29 22:11:42.902986 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:11:42.902215    4962 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.583s"
Oct 29 22:11:42.962787 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:42.962693    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks" podUID="a85f7d35-4bff-45d0-a392-2ecf056c2051" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:42.993840 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:42.993787    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:11:43.265447 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:43.253654    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:43.297534 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:43.267465    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg" podUID="70e1bafe-4b93-44d7-a179-44d5a82fd069" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:43.387548 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:43.387499    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:43.388894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:43.388859    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:43.630340 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:43.630066    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:11:43.745316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:43.721119    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:43.770312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:43.763494    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:11:43.875967 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:43.875832    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:11:43.911724 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:43.887592    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:11:44.081479 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.081432    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:44.168976 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.168864    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:11:44.301351 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.273554    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:11:44.301351 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.273638    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:11:44.301351 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.172459    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:11:44.301351 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.274226    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:11:44.301351 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.278763    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:11:44.301351 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.278814    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9" podUID="04500ffa-4da1-4d50-8fee-4e00ad03f4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 29 22:11:44.303859 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.303810    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:44.343534 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.343492    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded"
Oct 29 22:11:44.428369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.405859    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:44.441318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.437587    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded"
Oct 29 22:11:44.445309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.443550    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:11:44.449315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.447369    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:11:44.449315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.447456    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:11:44.519239 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.519208    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:11:44.519470 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.519444    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:11:44.549467 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.530366    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:44.661369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.653402    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:11:44.661369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.653463    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9" podUID="04500ffa-4da1-4d50-8fee-4e00ad03f4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 29 22:11:44.718500 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.718397    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:44.733509 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.733464    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:44.784744 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.755383    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:11:44.784744 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.755443    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr" podUID="10c5ec48-f48a-4703-a82c-9a6995506125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:44.794894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.794838    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:44.857358 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:11:44.836441    4962 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.582s"
Oct 29 22:11:44.857358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.836535    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 22:11:44.857358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.836625    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 22:11:44.857358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.836653    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw" event={"ID":"e8c4fd64-8818-475c-b3d0-579054bcd76a","Type":"ContainerStarted","Data":"4f352fd661b49576fdc341dc0b51b8669ec6f42f09978d609d3344d78824e3d0"}
Oct 29 22:11:44.857358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.836689    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:44.857358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.836718    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq"]
Oct 29 22:11:44.857358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.836754    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:44.857358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.836785    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"]
Oct 29 22:11:44.857358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.836812    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"]
Oct 29 22:11:44.880599 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.871397    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq" podUID="7224ed4a-e124-423c-b518-35c09cbc4383" containerName="maas-api" containerID="cri-o://7faf39cc8dd034073f098cb73b95e38193d1f138c34a82c409b333c5250d1dcb" gracePeriod=30
Oct 29 22:11:44.931757 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.913085    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:11:44.931757 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:44.913471    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:11:45.095570 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:45.091610    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:45.095570 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:45.091853    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:45.328334 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:45.319367    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:45.328964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:45.328930    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:11:45.329137 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:45.329112    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6" podUID="64a76b6c-1542-4fb1-9730-7b770c452b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:45.497312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:45.463114    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:45.503480 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:45.503438    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s" event={"ID":"b4cda533-309f-46e1-8cad-7239630ac19d","Type":"ContainerStarted","Data":"d86bfc8e1ad656b6fa7b688132a657e393a4e31b7a444aad34ec8c5d946ad206"}
Oct 29 22:11:45.605497 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:45.540511    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:45.605497 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:45.551456    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:11:45.605497 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:45.559428    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:11:45.615311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:45.605936    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:11:45.615311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:45.605999    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9" podUID="04500ffa-4da1-4d50-8fee-4e00ad03f4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 29 22:11:45.760382 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:45.757467    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:45.760382 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:45.757776    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:45.819333 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:45.795437    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:45.935453 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:45.935404    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:11:45.937686 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:45.937654    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:11:46.135365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:46.130716    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:11:46.201327 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:11:46.187929    4962 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7224ed4a_e124_423c_b518_35c09cbc4383.slice/crio-7faf39cc8dd034073f098cb73b95e38193d1f138c34a82c409b333c5250d1dcb.scope\": RecentStats: unable to find data in memory cache]"
Oct 29 22:11:46.350365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:46.344022    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:46.476843 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:46.476457    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:11:46.476843 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:46.476514    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9" podUID="04500ffa-4da1-4d50-8fee-4e00ad03f4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 29 22:11:46.515567 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:46.511041    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:11:46.515567 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:46.512069    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:11:46.515567 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:46.512120    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:11:46.543259 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:46.543218    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:46.667313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:46.649492    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:46.761441 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:46.761393    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:46.762112 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:46.761689    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:46.803102 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:46.802716    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:46.941317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:46.938082    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:46.941317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:46.939086    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:11:47.322733 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:47.322683    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:11:47.368314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:47.364461    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:47.458739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:47.449580    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:11:47.458739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:47.449642    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z" podUID="132ed775-c9e6-41f1-b224-fff550c6e073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:47.507313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:47.487939    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:11:47.533064 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:47.533023    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:11:47.537493 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:47.525625    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:47.538026 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:47.527557    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:11:47.538243 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:47.538215    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9" podUID="04500ffa-4da1-4d50-8fee-4e00ad03f4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 29 22:11:47.581762 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:47.571748    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:47.763682 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:47.763635    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:47.858500 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:47.836522    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:11:47.858500 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:47.836969    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:47.882311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:47.878395    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:47.964312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:47.939130    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:48.576310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:47.945990    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:48.576310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:48.482846    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:11:48.576310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:48.569429    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:11:48.576310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:48.569523    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:11:48.576310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:48.570152    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:11:48.576310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:48.570208    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc" podUID="036f71f6-61b8-469d-a080-d1ab0b45c195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 29 22:11:48.576310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:48.574425    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": context deadline exceeded"
Oct 29 22:11:48.674044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:48.661547    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 29 22:11:48.674044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:48.661639    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42" podUID="e1897161-0016-400a-9928-8b407f4148e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 29 22:11:48.715309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:48.698386    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:11:48.947377 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:48.923184    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:49.151632 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:49.144907    4962 generic.go:356] "Generic (PLEG): container finished" podID="7224ed4a-e124-423c-b518-35c09cbc4383" containerID="7faf39cc8dd034073f098cb73b95e38193d1f138c34a82c409b333c5250d1dcb" exitCode=0
Oct 29 22:11:49.172313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:49.170192    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq" event={"ID":"7224ed4a-e124-423c-b518-35c09cbc4383","Type":"ContainerDied","Data":"7faf39cc8dd034073f098cb73b95e38193d1f138c34a82c409b333c5250d1dcb"}
Oct 29 22:11:49.434626 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:49.434573    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": context deadline exceeded"
Oct 29 22:11:49.550601 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:49.550501    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:11:49.564482 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:49.558230    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:49.587476 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:49.587413    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:11:49.613373 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:49.590002    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:11:49.613667 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:49.613635    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:11:49.613804 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:49.598388    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:11:49.622685 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:49.622647    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:49.682309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:49.679391    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:11:49.700211 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:49.700172    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body=
Oct 29 22:11:49.777962 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:49.777907    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9" podUID="04500ffa-4da1-4d50-8fee-4e00ad03f4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused"
Oct 29 22:11:49.966349 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:49.950229    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:49.966349 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:49.957027    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:50.041807 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:50.005624    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw" event={"ID":"e8c4fd64-8818-475c-b3d0-579054bcd76a","Type":"ContainerStarted","Data":"5deab39450f38e650d846ff0ebd3c07ff1e0c246b2c66d6e46784f74bf0818d3"}
Oct 29 22:11:50.041807 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:50.012644    4962 generic.go:356] "Generic (PLEG): container finished" podID="036f71f6-61b8-469d-a080-d1ab0b45c195" containerID="271b7b89784cda842fead66ac67418b72e6abd7145340569ffe392ef431c8b62" exitCode=0
Oct 29 22:11:50.041807 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:50.012921    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc" event={"ID":"036f71f6-61b8-469d-a080-d1ab0b45c195","Type":"ContainerDied","Data":"271b7b89784cda842fead66ac67418b72e6abd7145340569ffe392ef431c8b62"}
Oct 29 22:11:50.041807 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:50.014642    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:11:50.074575 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:50.047034    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq"
Oct 29 22:11:50.289974 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:50.132463    4962 patch_prober.go:29] interesting pod/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:11:50.289974 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:50.132547    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc" podUID="35c2b463-e2a6-4b1d-8425-558ca7832668" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:50.995022 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:50.323876    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:50.995022 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:50.324597    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 29 22:11:50.995022 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:50.324660    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw" podUID="e8c4fd64-8818-475c-b3d0-579054bcd76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 29 22:11:50.995022 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:50.611042    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:50.995022 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:50.611379    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:50.995022 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:50.611502    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:11:50.995022 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:50.595403    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:50.995022 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:50.618325    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:50.995022 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:50.660481    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d" podUID="696bd863-08fa-4694-843f-ab65f3ccfeed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:50.995022 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:50.816907    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg" podUID="70e1bafe-4b93-44d7-a179-44d5a82fd069" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:50.995022 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:50.892792    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:50.995022 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:50.918418    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:11:50.995022 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:50.980027    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:51.031317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.023737    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:11:51.085721 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.085682    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:51.085973 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.029367    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:51.086459 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.032508    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 29 22:11:51.112149 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.112093    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s" podUID="b4cda533-309f-46e1-8cad-7239630ac19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 29 22:11:51.112705 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.025475    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:51.113133 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.062602    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body=
Oct 29 22:11:51.113342 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.113304    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused"
Oct 29 22:11:51.114027 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.114003    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 29 22:11:51.114178 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.114158    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s" podUID="b4cda533-309f-46e1-8cad-7239630ac19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 29 22:11:51.250613 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.246125    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw" podStartSLOduration=18.246091733 podStartE2EDuration="18.246091733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:50.946342037 +0000 UTC m=+1532.019299095" watchObservedRunningTime="2026-10-29 22:11:51.246091733 +0000 UTC m=+1532.319048698"
Oct 29 22:11:51.415312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.302397    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks" podUID="a85f7d35-4bff-45d0-a392-2ecf056c2051" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:51.415312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.389101    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 29 22:11:51.415312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.408398    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw" podUID="e8c4fd64-8818-475c-b3d0-579054bcd76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 29 22:11:51.619349 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.619234    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg" podUID="70e1bafe-4b93-44d7-a179-44d5a82fd069" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": context deadline exceeded"
Oct 29 22:11:51.682333 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.643488    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:51.682333 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.665525    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d" podUID="696bd863-08fa-4694-843f-ab65f3ccfeed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:51.706147 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.665722    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:11:52.092315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.665748    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:11:52.184090 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.732627    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7224ed4a-e124-423c-b518-35c09cbc4383-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7224ed4a-e124-423c-b518-35c09cbc4383-openshift-service-ca\") pod \"7224ed4a-e124-423c-b518-35c09cbc4383\" (UID: \"7224ed4a-e124-423c-b518-35c09cbc4383\") "
Oct 29 22:11:52.184335 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.758685    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s" podStartSLOduration=18.758663841 podStartE2EDuration="18.758663841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:51.411443934 +0000 UTC m=+1532.484400899" watchObservedRunningTime="2026-10-29 22:11:51.758663841 +0000 UTC m=+1532.831620809"
Oct 29 22:11:52.258430 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.783677    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": context deadline exceeded"
Oct 29 22:11:52.258430 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.825221    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7224ed4a-e124-423c-b518-35c09cbc4383-openshift-service-ca" pod "7224ed4a-e124-423c-b518-35c09cbc4383" (UID: "7224ed4a-e124-423c-b518-35c09cbc4383"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:52.258430 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:52.018167    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 29 22:11:52.258430 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:52.213354    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s" podUID="b4cda533-309f-46e1-8cad-7239630ac19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 29 22:11:52.258430 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.970033    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:52.258430 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.971554    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": context deadline exceeded"
Oct 29 22:11:52.258430 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.855752    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 22:11:52.258430 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:52.100958    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:11:52.258430 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.912809    4962 generic.go:356] "Generic (PLEG): container finished" podID="35c2b463-e2a6-4b1d-8425-558ca7832668" containerID="ee42b17744a6a1fc61efde1f2211e522f8707a0f8a5b4888967a0acae6e3ab33" exitCode=0
Oct 29 22:11:52.258430 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:52.114480    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:52.258430 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:51.972391    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:11:52.258430 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:52.101249    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:11:52.258430 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:52.114529    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:52.338642 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:52.307445    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks" podUID="a85f7d35-4bff-45d0-a392-2ecf056c2051" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:52.373130 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:52.372999    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 29 22:11:52.391689 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:52.391638    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw" podUID="e8c4fd64-8818-475c-b3d0-579054bcd76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 29 22:11:52.489351 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:52.479463    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7224ed4a-e124-423c-b518-35c09cbc4383-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7224ed4a-e124-423c-b518-35c09cbc4383-metrics-certs\") pod \"7224ed4a-e124-423c-b518-35c09cbc4383\" (UID: \"7224ed4a-e124-423c-b518-35c09cbc4383\") "
Oct 29 22:11:52.583313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:52.493825    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded"
Oct 29 22:11:52.583313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:52.561378    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq" podUID="7224ed4a-e124-423c-b518-35c09cbc4383" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.69:8443/health\": context deadline exceeded"
Oct 29 22:11:52.671600 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:52.671560    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg" podUID="70e1bafe-4b93-44d7-a179-44d5a82fd069" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:52.783666 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:52.783626    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7224ed4a-e124-423c-b518-35c09cbc4383-kube-api-access-2q4fr\" (UniqueName: \"kubernetes.io/projected/7224ed4a-e124-423c-b518-35c09cbc4383-kube-api-access-2q4fr\") pod \"7224ed4a-e124-423c-b518-35c09cbc4383\" (UID: \"7224ed4a-e124-423c-b518-35c09cbc4383\") "
Oct 29 22:11:53.063732 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.063641    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-envoy\") pod \"35c2b463-e2a6-4b1d-8425-558ca7832668\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") "
Oct 29 22:11:53.084314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.063986    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c2b463-e2a6-4b1d-8425-558ca7832668-kube-api-access-wsjfc\" (UniqueName: \"kubernetes.io/projected/35c2b463-e2a6-4b1d-8425-558ca7832668-kube-api-access-wsjfc\") pod \"35c2b463-e2a6-4b1d-8425-558ca7832668\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") "
Oct 29 22:11:53.084742 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.084713    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7224ed4a-e124-423c-b518-35c09cbc4383-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7224ed4a-e124-423c-b518-35c09cbc4383-maas-api-tls\") pod \"7224ed4a-e124-423c-b518-35c09cbc4383\" (UID: \"7224ed4a-e124-423c-b518-35c09cbc4383\") "
Oct 29 22:11:53.085021 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.085000    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-podinfo\") pod \"35c2b463-e2a6-4b1d-8425-558ca7832668\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") "
Oct 29 22:11:53.085170 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.085155    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-data\") pod \"35c2b463-e2a6-4b1d-8425-558ca7832668\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") "
Oct 29 22:11:53.104799 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:52.883099    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded"
Oct 29 22:11:53.104799 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.006667    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 22:11:53.104799 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:52.952523    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:11:53.137312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.119257    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:11:53.206933 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:11:53.194359    4962 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.091s"
Oct 29 22:11:53.206933 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.198485    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc" event={"ID":"35c2b463-e2a6-4b1d-8425-558ca7832668","Type":"ContainerDied","Data":"ee42b17744a6a1fc61efde1f2211e522f8707a0f8a5b4888967a0acae6e3ab33"}
Oct 29 22:11:53.230467 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.216687    4962 scope.go:122] "RemoveContainer" containerID="ee42b17744a6a1fc61efde1f2211e522f8707a0f8a5b4888967a0acae6e3ab33"
Oct 29 22:11:53.230467 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:52.952565    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d" podUID="696bd863-08fa-4694-843f-ab65f3ccfeed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded"
Oct 29 22:11:53.257315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.245526    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:11:53.257315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.246486    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:53.259391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.259353    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:53.261384 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.261352    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:53.333721 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.036671    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 29 22:11:53.333972 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.333943    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s" podUID="b4cda533-309f-46e1-8cad-7239630ac19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 29 22:11:53.334075 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.281462    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-data" pod "35c2b463-e2a6-4b1d-8425-558ca7832668" (UID: "35c2b463-e2a6-4b1d-8425-558ca7832668"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:53.334168 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.239402    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:11:53.368481 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.339972    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 29 22:11:53.368481 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.340017    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw" podUID="e8c4fd64-8818-475c-b3d0-579054bcd76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 29 22:11:53.368481 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.346358    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:11:53.450320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.420272    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:53.490327 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.490260    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7224ed4a-e124-423c-b518-35c09cbc4383-metrics-certs" pod "7224ed4a-e124-423c-b518-35c09cbc4383" (UID: "7224ed4a-e124-423c-b518-35c09cbc4383"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:11:53.528320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.524252    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-podinfo" pod "35c2b463-e2a6-4b1d-8425-558ca7832668" (UID: "35c2b463-e2a6-4b1d-8425-558ca7832668"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:11:53.601363 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.587872    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7224ed4a-e124-423c-b518-35c09cbc4383-maas-api-tls" pod "7224ed4a-e124-423c-b518-35c09cbc4383" (UID: "7224ed4a-e124-423c-b518-35c09cbc4383"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:11:53.601363 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.588147    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c2b463-e2a6-4b1d-8425-558ca7832668-kube-api-access-wsjfc" pod "35c2b463-e2a6-4b1d-8425-558ca7832668" (UID: "35c2b463-e2a6-4b1d-8425-558ca7832668"). InnerVolumeSpecName "kube-api-access-wsjfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:53.605001 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.604969    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-envoy" pod "35c2b463-e2a6-4b1d-8425-558ca7832668" (UID: "35c2b463-e2a6-4b1d-8425-558ca7832668"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:53.634311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.619040    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsjfc\" (UniqueName: \"kubernetes.io/projected/35c2b463-e2a6-4b1d-8425-558ca7832668-kube-api-access-wsjfc\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:53.681874 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.681841    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7224ed4a-e124-423c-b518-35c09cbc4383-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:53.682089 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.682076    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:53.682185 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.682173    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:53.697375 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.697343    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7224ed4a-e124-423c-b518-35c09cbc4383-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:53.789714 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.697591    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7224ed4a-e124-423c-b518-35c09cbc4383-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:54.096493 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:53.911485    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7224ed4a-e124-423c-b518-35c09cbc4383-kube-api-access-2q4fr" pod "7224ed4a-e124-423c-b518-35c09cbc4383" (UID: "7224ed4a-e124-423c-b518-35c09cbc4383"). InnerVolumeSpecName "kube-api-access-2q4fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:54.107318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.023538    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 29 22:11:54.143800 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.143758    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s" podUID="b4cda533-309f-46e1-8cad-7239630ac19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 29 22:11:54.143972 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.142396    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:54.180218 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.178328    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:11:54.198362 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.195256    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:11:54.198362 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.195392    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2"]
Oct 29 22:11:54.222739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.203955    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-workload-certs\") pod \"35c2b463-e2a6-4b1d-8425-558ca7832668\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") "
Oct 29 22:11:54.222739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.204015    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-token\" (UniqueName: \"kubernetes.io/projected/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-token\") pod \"35c2b463-e2a6-4b1d-8425-558ca7832668\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") "
Oct 29 22:11:54.222739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.204083    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-podinfo\") pod \"036f71f6-61b8-469d-a080-d1ab0b45c195\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") "
Oct 29 22:11:54.222739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.204211    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/036f71f6-61b8-469d-a080-d1ab0b45c195-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036f71f6-61b8-469d-a080-d1ab0b45c195-istiod-ca-cert\") pod \"036f71f6-61b8-469d-a080-d1ab0b45c195\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") "
Oct 29 22:11:54.222739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.204242    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/036f71f6-61b8-469d-a080-d1ab0b45c195-kube-api-access-rdvjw\" (UniqueName: \"kubernetes.io/projected/036f71f6-61b8-469d-a080-d1ab0b45c195-kube-api-access-rdvjw\") pod \"036f71f6-61b8-469d-a080-d1ab0b45c195\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") "
Oct 29 22:11:54.222739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.204311    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-envoy\") pod \"036f71f6-61b8-469d-a080-d1ab0b45c195\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") "
Oct 29 22:11:54.222739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.204466    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-workload-certs\") pod \"036f71f6-61b8-469d-a080-d1ab0b45c195\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") "
Oct 29 22:11:54.222739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.204552    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-token\" (UniqueName: \"kubernetes.io/projected/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-token\") pod \"036f71f6-61b8-469d-a080-d1ab0b45c195\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") "
Oct 29 22:11:54.222739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.204659    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-workload-socket\") pod \"036f71f6-61b8-469d-a080-d1ab0b45c195\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") "
Oct 29 22:11:54.222739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.204801    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-data\") pod \"036f71f6-61b8-469d-a080-d1ab0b45c195\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") "
Oct 29 22:11:54.222739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.204866    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-workload-socket\") pod \"35c2b463-e2a6-4b1d-8425-558ca7832668\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") "
Oct 29 22:11:54.222739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.204902    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-credential-socket\") pod \"35c2b463-e2a6-4b1d-8425-558ca7832668\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") "
Oct 29 22:11:54.222739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.204932    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-credential-socket\") pod \"036f71f6-61b8-469d-a080-d1ab0b45c195\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") "
Oct 29 22:11:54.222739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.205005    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35c2b463-e2a6-4b1d-8425-558ca7832668-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c2b463-e2a6-4b1d-8425-558ca7832668-istiod-ca-cert\") pod \"35c2b463-e2a6-4b1d-8425-558ca7832668\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") "
Oct 29 22:11:54.222739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.206166    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q4fr\" (UniqueName: \"kubernetes.io/projected/7224ed4a-e124-423c-b518-35c09cbc4383-kube-api-access-2q4fr\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:54.222739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.206189    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:54.239089 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.224627    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 29 22:11:54.239089 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.224684    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw" podUID="e8c4fd64-8818-475c-b3d0-579054bcd76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 29 22:11:54.239089 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.227243    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-workload-socket" pod "35c2b463-e2a6-4b1d-8425-558ca7832668" (UID: "35c2b463-e2a6-4b1d-8425-558ca7832668"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:54.239089 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.227828    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-credential-socket" pod "35c2b463-e2a6-4b1d-8425-558ca7832668" (UID: "35c2b463-e2a6-4b1d-8425-558ca7832668"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:54.239089 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.228056    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-credential-socket" pod "036f71f6-61b8-469d-a080-d1ab0b45c195" (UID: "036f71f6-61b8-469d-a080-d1ab0b45c195"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:54.240689 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.240656    4962 cpu_manager.go:414] "containerMap: removing container" podUID="036f71f6-61b8-469d-a080-d1ab0b45c195" containerName="istio-proxy"
Oct 29 22:11:54.240862 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.240848    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="036f71f6-61b8-469d-a080-d1ab0b45c195" containerName="istio-proxy"
Oct 29 22:11:54.336353 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.336324    4962 cpu_manager.go:414] "containerMap: removing container" podUID="35c2b463-e2a6-4b1d-8425-558ca7832668" containerName="istio-proxy"
Oct 29 22:11:54.386052 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.386018    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35c2b463-e2a6-4b1d-8425-558ca7832668" containerName="istio-proxy"
Oct 29 22:11:54.386388 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.386352    4962 cpu_manager.go:414] "containerMap: removing container" podUID="7224ed4a-e124-423c-b518-35c09cbc4383" containerName="maas-api"
Oct 29 22:11:54.386597 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.386577    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7224ed4a-e124-423c-b518-35c09cbc4383" containerName="maas-api"
Oct 29 22:11:54.387249 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.387230    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="35c2b463-e2a6-4b1d-8425-558ca7832668" containerName="istio-proxy"
Oct 29 22:11:54.387470 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.387449    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="7224ed4a-e124-423c-b518-35c09cbc4383" containerName="maas-api"
Oct 29 22:11:54.387670 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.387626    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="036f71f6-61b8-469d-a080-d1ab0b45c195" containerName="istio-proxy"
Oct 29 22:11:54.673444 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.241094    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-workload-certs" pod "35c2b463-e2a6-4b1d-8425-558ca7832668" (UID: "35c2b463-e2a6-4b1d-8425-558ca7832668"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:54.673689 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.335405    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-workload-certs" pod "036f71f6-61b8-469d-a080-d1ab0b45c195" (UID: "036f71f6-61b8-469d-a080-d1ab0b45c195"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:54.673783 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.342574    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c2b463-e2a6-4b1d-8425-558ca7832668-istiod-ca-cert" pod "35c2b463-e2a6-4b1d-8425-558ca7832668" (UID: "35c2b463-e2a6-4b1d-8425-558ca7832668"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:54.673853 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.482428    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-data" pod "036f71f6-61b8-469d-a080-d1ab0b45c195" (UID: "036f71f6-61b8-469d-a080-d1ab0b45c195"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:54.673926 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.486395    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-workload-socket" pod "036f71f6-61b8-469d-a080-d1ab0b45c195" (UID: "036f71f6-61b8-469d-a080-d1ab0b45c195"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:54.674014 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.427548    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:11:54.698350 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.394241    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:54.725669 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.509369    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:11:54.727030 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.509408    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": context deadline exceeded"
Oct 29 22:11:54.893201 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.510317    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq"
Oct 29 22:11:55.014412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.512366    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:55.014412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.692925    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:55.014412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.000459    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq" event={"ID":"7224ed4a-e124-423c-b518-35c09cbc4383","Type":"ContainerDied","Data":"8ee038754cb11a7acde6b8134f599cef03f24d5ffa8264166d5dfe7ae813c51d"}
Oct 29 22:11:55.014412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.000521    4962 scope.go:122] "RemoveContainer" containerID="7faf39cc8dd034073f098cb73b95e38193d1f138c34a82c409b333c5250d1dcb"
Oct 29 22:11:55.014412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.427508    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:55.014412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.394278    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded"
Oct 29 22:11:55.014412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.394316    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:55.014412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.468617    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": context deadline exceeded"
Oct 29 22:11:55.014412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.693551    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2"
Oct 29 22:11:55.014412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.747538    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/036f71f6-61b8-469d-a080-d1ab0b45c195-istiod-ca-cert" pod "036f71f6-61b8-469d-a080-d1ab0b45c195" (UID: "036f71f6-61b8-469d-a080-d1ab0b45c195"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:55.014412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.942389    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-token" pod "036f71f6-61b8-469d-a080-d1ab0b45c195" (UID: "036f71f6-61b8-469d-a080-d1ab0b45c195"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:55.014412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.942538    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-envoy" pod "036f71f6-61b8-469d-a080-d1ab0b45c195" (UID: "036f71f6-61b8-469d-a080-d1ab0b45c195"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:55.014412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:54.949513    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-podinfo" pod "036f71f6-61b8-469d-a080-d1ab0b45c195" (UID: "036f71f6-61b8-469d-a080-d1ab0b45c195"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:11:55.014412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.000914    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-token" pod "35c2b463-e2a6-4b1d-8425-558ca7832668" (UID: "35c2b463-e2a6-4b1d-8425-558ca7832668"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:55.019281 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.019252    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:55.027275 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.027245    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:55.027558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.027530    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc" event={"ID":"036f71f6-61b8-469d-a080-d1ab0b45c195","Type":"ContainerDied","Data":"1b57193608247cac8cff015727aefd501911084bb8016f4ad4ba21747221e737"}
Oct 29 22:11:55.027710 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.027692    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2"]
Oct 29 22:11:55.027838 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.027827    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"]
Oct 29 22:11:55.028145 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.028126    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"
Oct 29 22:11:55.028346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.028331    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 22:11:55.030133 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.029987    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 22:11:55.030341 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.030316    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"]
Oct 29 22:11:55.030580 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.030564    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-xnbks"
Oct 29 22:11:55.030986 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.030938    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 22:11:55.031191 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.031176    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg"
Oct 29 22:11:55.031480 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.031452    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp" podUID="5d5e71cf-5078-4917-94b8-2b531b143393" containerName="istio-proxy" containerID="cri-o://3c7774bee68cad1eae65e008896454a04da85c5ea27d3083c5f4af47e55854ad" gracePeriod=30
Oct 29 22:11:55.031730 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.031712    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d"
Oct 29 22:11:55.033348 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.033324    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv" podUID="d71de0ec-1eaa-481f-8cee-cb256e330136" containerName="istio-proxy" containerID="cri-o://c7bbd6fb4c953b189e12f0cbdc8d2640a8a4666d5e467a6b82d2253dd3127a19" gracePeriod=30
Oct 29 22:11:55.048306 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.044471    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-75db5b\""
Oct 29 22:11:55.048306 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.044807    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-envoy\") pod \"036f71f6-61b8-469d-a080-d1ab0b45c195\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") "
Oct 29 22:11:55.048306 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:11:55.044882    4962 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/036f71f6-61b8-469d-a080-d1ab0b45c195/volumes/kubernetes.io~empty-dir/istio-envoy
Oct 29 22:11:55.048306 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.044897    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-envoy" pod "036f71f6-61b8-469d-a080-d1ab0b45c195" (UID: "036f71f6-61b8-469d-a080-d1ab0b45c195"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:55.083391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.055587    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 29 22:11:55.083391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.055636    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s" podUID="b4cda533-309f-46e1-8cad-7239630ac19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 29 22:11:55.083391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.068729    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-workload-certs\") pod \"036f71f6-61b8-469d-a080-d1ab0b45c195\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") "
Oct 29 22:11:55.083391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.068807    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-token\" (UniqueName: \"kubernetes.io/projected/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-token\") pod \"036f71f6-61b8-469d-a080-d1ab0b45c195\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") "
Oct 29 22:11:55.083391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.068902    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-workload-socket\") pod \"036f71f6-61b8-469d-a080-d1ab0b45c195\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") "
Oct 29 22:11:55.083391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.069005    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-data\") pod \"036f71f6-61b8-469d-a080-d1ab0b45c195\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") "
Oct 29 22:11:55.083391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.069078    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35c2b463-e2a6-4b1d-8425-558ca7832668-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c2b463-e2a6-4b1d-8425-558ca7832668-istiod-ca-cert\") pod \"35c2b463-e2a6-4b1d-8425-558ca7832668\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") "
Oct 29 22:11:55.083391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.069120    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-workload-certs\") pod \"35c2b463-e2a6-4b1d-8425-558ca7832668\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") "
Oct 29 22:11:55.083391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.069156    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-token\" (UniqueName: \"kubernetes.io/projected/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-token\") pod \"35c2b463-e2a6-4b1d-8425-558ca7832668\" (UID: \"35c2b463-e2a6-4b1d-8425-558ca7832668\") "
Oct 29 22:11:55.083391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.069208    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-podinfo\") pod \"036f71f6-61b8-469d-a080-d1ab0b45c195\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") "
Oct 29 22:11:55.083391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.069387    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/036f71f6-61b8-469d-a080-d1ab0b45c195-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036f71f6-61b8-469d-a080-d1ab0b45c195-istiod-ca-cert\") pod \"036f71f6-61b8-469d-a080-d1ab0b45c195\" (UID: \"036f71f6-61b8-469d-a080-d1ab0b45c195\") "
Oct 29 22:11:55.083391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.070192    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.083391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.070213    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.083391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.070229    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.083391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.070245    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.083391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.071064    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036f71f6-61b8-469d-a080-d1ab0b45c195-kube-api-access-rdvjw" pod "036f71f6-61b8-469d-a080-d1ab0b45c195" (UID: "036f71f6-61b8-469d-a080-d1ab0b45c195"). InnerVolumeSpecName "kube-api-access-rdvjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:55.083391 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:11:55.071350    4962 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/036f71f6-61b8-469d-a080-d1ab0b45c195/volumes/kubernetes.io~empty-dir/workload-certs
Oct 29 22:11:55.083391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.071365    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-workload-certs" pod "036f71f6-61b8-469d-a080-d1ab0b45c195" (UID: "036f71f6-61b8-469d-a080-d1ab0b45c195"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:55.085416 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:11:55.085378    4962 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/036f71f6-61b8-469d-a080-d1ab0b45c195/volumes/kubernetes.io~projected/istio-token
Oct 29 22:11:55.085712 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.085615    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-token" pod "036f71f6-61b8-469d-a080-d1ab0b45c195" (UID: "036f71f6-61b8-469d-a080-d1ab0b45c195"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:55.085924 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:11:55.085863    4962 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/036f71f6-61b8-469d-a080-d1ab0b45c195/volumes/kubernetes.io~empty-dir/workload-socket
Oct 29 22:11:55.085924 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.085882    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-workload-socket" pod "036f71f6-61b8-469d-a080-d1ab0b45c195" (UID: "036f71f6-61b8-469d-a080-d1ab0b45c195"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:55.087906 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:11:55.087873    4962 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/036f71f6-61b8-469d-a080-d1ab0b45c195/volumes/kubernetes.io~empty-dir/istio-data
Oct 29 22:11:55.088115 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.088045    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-data" pod "036f71f6-61b8-469d-a080-d1ab0b45c195" (UID: "036f71f6-61b8-469d-a080-d1ab0b45c195"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:55.088306 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:11:55.088262    4962 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/35c2b463-e2a6-4b1d-8425-558ca7832668/volumes/kubernetes.io~configmap/istiod-ca-cert
Oct 29 22:11:55.088474 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.088278    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c2b463-e2a6-4b1d-8425-558ca7832668-istiod-ca-cert" pod "35c2b463-e2a6-4b1d-8425-558ca7832668" (UID: "35c2b463-e2a6-4b1d-8425-558ca7832668"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:55.088649 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:11:55.088454    4962 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/35c2b463-e2a6-4b1d-8425-558ca7832668/volumes/kubernetes.io~empty-dir/workload-certs
Oct 29 22:11:55.088649 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.088583    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-workload-certs" pod "35c2b463-e2a6-4b1d-8425-558ca7832668" (UID: "35c2b463-e2a6-4b1d-8425-558ca7832668"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:55.088898 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:11:55.088858    4962 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/35c2b463-e2a6-4b1d-8425-558ca7832668/volumes/kubernetes.io~projected/istio-token
Oct 29 22:11:55.089083 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.089047    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-token" pod "35c2b463-e2a6-4b1d-8425-558ca7832668" (UID: "35c2b463-e2a6-4b1d-8425-558ca7832668"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:55.089305 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:11:55.089257    4962 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/036f71f6-61b8-469d-a080-d1ab0b45c195/volumes/kubernetes.io~downward-api/istio-podinfo
Oct 29 22:11:55.089486 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.089274    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-podinfo" pod "036f71f6-61b8-469d-a080-d1ab0b45c195" (UID: "036f71f6-61b8-469d-a080-d1ab0b45c195"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:11:55.089664 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:11:55.089468    4962 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/036f71f6-61b8-469d-a080-d1ab0b45c195/volumes/kubernetes.io~configmap/istiod-ca-cert
Oct 29 22:11:55.089664 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.089604    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/036f71f6-61b8-469d-a080-d1ab0b45c195-istiod-ca-cert" pod "036f71f6-61b8-469d-a080-d1ab0b45c195" (UID: "036f71f6-61b8-469d-a080-d1ab0b45c195"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:55.096364 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.096344    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:11:55.096914 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.096893    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-75db5b-dockercfg-pssln\""
Oct 29 22:11:55.153756 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.142799    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 22:11:55.154413 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.154388    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:11:55.154631 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.154616    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:11:55.154771 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.154759    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 22:11:55.183668 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.183632    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s" event={"ID":"b4cda533-309f-46e1-8cad-7239630ac19d","Type":"ContainerStarted","Data":"04c3fdbbfbc59616646d2a8cfc2ead6b536ca689936ba6636e643a935c4478a5"}
Oct 29 22:11:55.201035 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.193189    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.201035 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.193222    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c2b463-e2a6-4b1d-8425-558ca7832668-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.201035 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.193237    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c2b463-e2a6-4b1d-8425-558ca7832668-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.201035 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.193249    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c2b463-e2a6-4b1d-8425-558ca7832668-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.201035 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.193262    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.201035 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.193274    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036f71f6-61b8-469d-a080-d1ab0b45c195-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.201035 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.193304    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdvjw\" (UniqueName: \"kubernetes.io/projected/036f71f6-61b8-469d-a080-d1ab0b45c195-kube-api-access-rdvjw\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.201035 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.193318    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.201035 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.193333    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036f71f6-61b8-469d-a080-d1ab0b45c195-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.201035 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.193346    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036f71f6-61b8-469d-a080-d1ab0b45c195-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.243355 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.241330    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:55.269323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.269251    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:11:55.287308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.284967    4962 scope.go:122] "RemoveContainer" containerID="271b7b89784cda842fead66ac67418b72e6abd7145340569ffe392ef431c8b62"
Oct 29 22:11:55.299306 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.296203    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1897161-0016-400a-9928-8b407f4148e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1897161-0016-400a-9928-8b407f4148e9-istiod-ca-cert\") pod \"e1897161-0016-400a-9928-8b407f4148e9\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") "
Oct 29 22:11:55.299306 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.296250    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-workload-socket\") pod \"e1897161-0016-400a-9928-8b407f4148e9\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") "
Oct 29 22:11:55.299306 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.296399    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-credential-socket\") pod \"e1897161-0016-400a-9928-8b407f4148e9\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") "
Oct 29 22:11:55.299306 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.296453    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-workload-certs\") pod \"e1897161-0016-400a-9928-8b407f4148e9\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") "
Oct 29 22:11:55.299306 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.296551    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-istio-envoy\") pod \"e1897161-0016-400a-9928-8b407f4148e9\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") "
Oct 29 22:11:55.299306 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.296605    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1897161-0016-400a-9928-8b407f4148e9-istio-token\" (UniqueName: \"kubernetes.io/projected/e1897161-0016-400a-9928-8b407f4148e9-istio-token\") pod \"e1897161-0016-400a-9928-8b407f4148e9\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") "
Oct 29 22:11:55.299306 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.296652    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1897161-0016-400a-9928-8b407f4148e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1897161-0016-400a-9928-8b407f4148e9-istio-podinfo\") pod \"e1897161-0016-400a-9928-8b407f4148e9\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") "
Oct 29 22:11:55.299306 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.296791    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1897161-0016-400a-9928-8b407f4148e9-kube-api-access-jzhzx\" (UniqueName: \"kubernetes.io/projected/e1897161-0016-400a-9928-8b407f4148e9-kube-api-access-jzhzx\") pod \"e1897161-0016-400a-9928-8b407f4148e9\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") "
Oct 29 22:11:55.299306 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.296932    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-istio-data\") pod \"e1897161-0016-400a-9928-8b407f4148e9\" (UID: \"e1897161-0016-400a-9928-8b407f4148e9\") "
Oct 29 22:11:55.299306 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.297188    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvdff\" (UniqueName: \"kubernetes.io/projected/6033cb02-b74f-4cfc-9073-646c4ea795cf-kube-api-access-xvdff\") pod \"payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2\" (UID: \"6033cb02-b74f-4cfc-9073-646c4ea795cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2"
Oct 29 22:11:55.307447 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.305061    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6033cb02-b74f-4cfc-9073-646c4ea795cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2\" (UID: \"6033cb02-b74f-4cfc-9073-646c4ea795cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2"
Oct 29 22:11:55.309314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.308864    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"]
Oct 29 22:11:55.309314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.309133    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57" podUID="67b4a685-bfa5-4a36-adf7-68c7591137c3" containerName="istio-proxy" containerID="cri-o://733129bde7f2b2532aaf9c3791293ff0e39979c9dafb1975650ec8f7f4bdd8fd" gracePeriod=30
Oct 29 22:11:55.312441 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.311052    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-workload-socket" pod "e1897161-0016-400a-9928-8b407f4148e9" (UID: "e1897161-0016-400a-9928-8b407f4148e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:55.312441 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.311356    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-credential-socket" pod "e1897161-0016-400a-9928-8b407f4148e9" (UID: "e1897161-0016-400a-9928-8b407f4148e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:55.312441 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.311598    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-workload-certs" pod "e1897161-0016-400a-9928-8b407f4148e9" (UID: "e1897161-0016-400a-9928-8b407f4148e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:55.327593 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.327567    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-istio-data" pod "e1897161-0016-400a-9928-8b407f4148e9" (UID: "e1897161-0016-400a-9928-8b407f4148e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:55.327928 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.327907    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 29 22:11:55.328078 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.328055    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw" podUID="e8c4fd64-8818-475c-b3d0-579054bcd76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 29 22:11:55.328612 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.328583    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc" event={"ID":"35c2b463-e2a6-4b1d-8425-558ca7832668","Type":"ContainerDied","Data":"6a803f26456110552424f7e4ef6beeaf8df1237237b58b4e034c66b702bf78ee"}
Oct 29 22:11:55.328712 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.328680    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"
Oct 29 22:11:55.339725 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.339699    4962 generic.go:356] "Generic (PLEG): container finished" podID="e1897161-0016-400a-9928-8b407f4148e9" containerID="2fb3d9576ee00f91c232b0db8a4d81c05a44ba966996f0907fb75967ce88eb72" exitCode=0
Oct 29 22:11:55.340426 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.340398    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42" event={"ID":"e1897161-0016-400a-9928-8b407f4148e9","Type":"ContainerDied","Data":"2fb3d9576ee00f91c232b0db8a4d81c05a44ba966996f0907fb75967ce88eb72"}
Oct 29 22:11:55.340557 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.340528    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1897161-0016-400a-9928-8b407f4148e9-istiod-ca-cert" pod "e1897161-0016-400a-9928-8b407f4148e9" (UID: "e1897161-0016-400a-9928-8b407f4148e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:11:55.340838 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.340708    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"
Oct 29 22:11:55.350939 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.350914    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1897161-0016-400a-9928-8b407f4148e9-istio-token" pod "e1897161-0016-400a-9928-8b407f4148e9" (UID: "e1897161-0016-400a-9928-8b407f4148e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:55.351581 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.351537    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-istio-envoy" pod "e1897161-0016-400a-9928-8b407f4148e9" (UID: "e1897161-0016-400a-9928-8b407f4148e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:11:55.352084 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.351928    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1897161-0016-400a-9928-8b407f4148e9-kube-api-access-jzhzx" pod "e1897161-0016-400a-9928-8b407f4148e9" (UID: "e1897161-0016-400a-9928-8b407f4148e9"). InnerVolumeSpecName "kube-api-access-jzhzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:11:55.352187 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.352163    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"]
Oct 29 22:11:55.352370 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.352335    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1897161-0016-400a-9928-8b407f4148e9-istio-podinfo" pod "e1897161-0016-400a-9928-8b407f4148e9" (UID: "e1897161-0016-400a-9928-8b407f4148e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:11:55.352453 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.352410    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf" podUID="84e0c5a1-5acf-4a80-8448-5aec9f52b0b8" containerName="istio-proxy" containerID="cri-o://b8938ecc7272a4a12de802fbe8b51101e19c7cf48dc91072654e0d3b6a0642ca" gracePeriod=30
Oct 29 22:11:55.399937 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.399910    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:11:55.421324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.420265    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvdff\" (UniqueName: \"kubernetes.io/projected/6033cb02-b74f-4cfc-9073-646c4ea795cf-kube-api-access-xvdff\") pod \"payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2\" (UID: \"6033cb02-b74f-4cfc-9073-646c4ea795cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2"
Oct 29 22:11:55.421324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.420444    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6033cb02-b74f-4cfc-9073-646c4ea795cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2\" (UID: \"6033cb02-b74f-4cfc-9073-646c4ea795cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2"
Oct 29 22:11:55.421324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.421001    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.421324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.421025    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1897161-0016-400a-9928-8b407f4148e9-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.421324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.421039    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.421324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.421052    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.421324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.421065    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.421324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.421079    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1897161-0016-400a-9928-8b407f4148e9-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.421324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.421091    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1897161-0016-400a-9928-8b407f4148e9-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.421324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.421103    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1897161-0016-400a-9928-8b407f4148e9-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.421324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.421116    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzhzx\" (UniqueName: \"kubernetes.io/projected/e1897161-0016-400a-9928-8b407f4148e9-kube-api-access-jzhzx\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:11:55.468073 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.467833    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6033cb02-b74f-4cfc-9073-646c4ea795cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2\" (UID: \"6033cb02-b74f-4cfc-9073-646c4ea795cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2"
Oct 29 22:11:55.486309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.482575    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44"]
Oct 29 22:11:55.486309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.483904    4962 cpu_manager.go:414] "containerMap: removing container" podUID="e1897161-0016-400a-9928-8b407f4148e9" containerName="istio-proxy"
Oct 29 22:11:55.486309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.483932    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1897161-0016-400a-9928-8b407f4148e9" containerName="istio-proxy"
Oct 29 22:11:55.486309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.484180    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1897161-0016-400a-9928-8b407f4148e9" containerName="istio-proxy"
Oct 29 22:11:55.501405 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.501375    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44"
Oct 29 22:11:55.540330 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.529992    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-4e8583-serving-cert\""
Oct 29 22:11:55.540330 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.532232    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvdff\" (UniqueName: \"kubernetes.io/projected/6033cb02-b74f-4cfc-9073-646c4ea795cf-kube-api-access-xvdff\") pod \"payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2\" (UID: \"6033cb02-b74f-4cfc-9073-646c4ea795cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2"
Oct 29 22:11:55.616754 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.590726    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44"]
Oct 29 22:11:55.663665 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.639368    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blfjw\" (UniqueName: \"kubernetes.io/projected/674cc938-504c-4547-abb7-cc2ee413d6cf-kube-api-access-blfjw\") pod \"maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44\" (UID: \"674cc938-504c-4547-abb7-cc2ee413d6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44"
Oct 29 22:11:55.663665 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.639609    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/674cc938-504c-4547-abb7-cc2ee413d6cf-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44\" (UID: \"674cc938-504c-4547-abb7-cc2ee413d6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44"
Oct 29 22:11:55.663665 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.639788    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/674cc938-504c-4547-abb7-cc2ee413d6cf-metrics-certs\") pod \"maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44\" (UID: \"674cc938-504c-4547-abb7-cc2ee413d6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44"
Oct 29 22:11:55.663665 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.639948    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/674cc938-504c-4547-abb7-cc2ee413d6cf-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44\" (UID: \"674cc938-504c-4547-abb7-cc2ee413d6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44"
Oct 29 22:11:55.701696 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.701557    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:11:55.733599 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.732155    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2"
Oct 29 22:11:55.744067 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.744034    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/674cc938-504c-4547-abb7-cc2ee413d6cf-metrics-certs\") pod \"maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44\" (UID: \"674cc938-504c-4547-abb7-cc2ee413d6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44"
Oct 29 22:11:55.744531 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.744509    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/674cc938-504c-4547-abb7-cc2ee413d6cf-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44\" (UID: \"674cc938-504c-4547-abb7-cc2ee413d6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44"
Oct 29 22:11:55.744691 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.744675    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blfjw\" (UniqueName: \"kubernetes.io/projected/674cc938-504c-4547-abb7-cc2ee413d6cf-kube-api-access-blfjw\") pod \"maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44\" (UID: \"674cc938-504c-4547-abb7-cc2ee413d6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44"
Oct 29 22:11:55.749481 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.747444    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/674cc938-504c-4547-abb7-cc2ee413d6cf-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44\" (UID: \"674cc938-504c-4547-abb7-cc2ee413d6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44"
Oct 29 22:11:55.749481 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.748268    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/674cc938-504c-4547-abb7-cc2ee413d6cf-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44\" (UID: \"674cc938-504c-4547-abb7-cc2ee413d6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44"
Oct 29 22:11:55.749481 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.748877    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/674cc938-504c-4547-abb7-cc2ee413d6cf-metrics-certs\") pod \"maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44\" (UID: \"674cc938-504c-4547-abb7-cc2ee413d6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44"
Oct 29 22:11:55.749481 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.748948    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:55.749481 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.749228    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:11:55.758449 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.754794    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/674cc938-504c-4547-abb7-cc2ee413d6cf-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44\" (UID: \"674cc938-504c-4547-abb7-cc2ee413d6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44"
Oct 29 22:11:55.846573 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.784145    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:11:55.846573 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.784445    4962 scope.go:122] "RemoveContainer" containerID="2fb3d9576ee00f91c232b0db8a4d81c05a44ba966996f0907fb75967ce88eb72"
Oct 29 22:11:55.846573 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.830008    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blfjw\" (UniqueName: \"kubernetes.io/projected/674cc938-504c-4547-abb7-cc2ee413d6cf-kube-api-access-blfjw\") pod \"maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44\" (UID: \"674cc938-504c-4547-abb7-cc2ee413d6cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44"
Oct 29 22:11:55.863744 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:55.863722    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 22:11:56.058406 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:56.057500    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 29 22:11:56.058406 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:56.057562    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s" podUID="b4cda533-309f-46e1-8cad-7239630ac19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 29 22:11:56.058692 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:56.058667    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:11:56.062033 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:56.061341    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 22:11:56.143430 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:56.143400    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44"
Oct 29 22:11:56.201652 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:56.199434    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 29 22:11:56.201652 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:56.199499    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw" podUID="e8c4fd64-8818-475c-b3d0-579054bcd76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 29 22:11:56.324753 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:11:56.324656    4962 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1897161_0016_400a_9928_8b407f4148e9.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7224ed4a_e124_423c_b518_35c09cbc4383.slice/crio-8ee038754cb11a7acde6b8134f599cef03f24d5ffa8264166d5dfe7ae813c51d\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7224ed4a_e124_423c_b518_35c09cbc4383.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod036f71f6_61b8_469d_a080_d1ab0b45c195.slice\": RecentStats: unable to find data in memory cache]"
Oct 29 22:11:56.383840 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:56.383800    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42" event={"ID":"e1897161-0016-400a-9928-8b407f4148e9","Type":"ContainerDied","Data":"4b653a55a89d52585ac1053c22a09803c1866666694976ef947d3918776881b9"}
Oct 29 22:11:56.479445 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:56.479360    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq"]
Oct 29 22:11:56.569883 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:56.568702    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-a9def7-85b6bf5745-jcdzq"]
Oct 29 22:11:56.643055 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:56.643026    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"]
Oct 29 22:11:56.652947 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:56.651245    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-ghl42"]
Oct 29 22:11:56.664903 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:56.664820    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"]
Oct 29 22:11:56.671024 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:56.671002    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-d7xnc"]
Oct 29 22:11:56.765132 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:56.763940    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2"]
Oct 29 22:11:56.942556 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:56.942522    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44"]
Oct 29 22:11:56.961156 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:56.961121    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"]
Oct 29 22:11:56.964780 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:56.964745    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a3c02e-openshift-default-65c6c99867-g85lc"]
Oct 29 22:11:56.998866 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:56.998825    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 29 22:11:56.999107 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:56.999081    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s" podUID="b4cda533-309f-46e1-8cad-7239630ac19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 29 22:11:57.224754 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:57.205098    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 29 22:11:57.224754 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:57.205155    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw" podUID="e8c4fd64-8818-475c-b3d0-579054bcd76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 29 22:11:57.619637 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:57.619557    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="036f71f6-61b8-469d-a080-d1ab0b45c195" path="/var/lib/kubelet/pods/036f71f6-61b8-469d-a080-d1ab0b45c195/volumes"
Oct 29 22:11:57.620671 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:57.620646    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35c2b463-e2a6-4b1d-8425-558ca7832668" path="/var/lib/kubelet/pods/35c2b463-e2a6-4b1d-8425-558ca7832668/volumes"
Oct 29 22:11:57.621561 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:57.621543    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7224ed4a-e124-423c-b518-35c09cbc4383" path="/var/lib/kubelet/pods/7224ed4a-e124-423c-b518-35c09cbc4383/volumes"
Oct 29 22:11:57.623058 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:57.623039    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1897161-0016-400a-9928-8b407f4148e9" path="/var/lib/kubelet/pods/e1897161-0016-400a-9928-8b407f4148e9/volumes"
Oct 29 22:11:57.633278 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:57.633259    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2" event={"ID":"6033cb02-b74f-4cfc-9073-646c4ea795cf","Type":"ContainerStarted","Data":"d2399bc9194752bfb7eaab93c62ab05edc073d29cc2195c9f4163334a7b7a772"}
Oct 29 22:11:57.633448 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:57.633432    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2"
Oct 29 22:11:57.633538 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:57.633526    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2" event={"ID":"6033cb02-b74f-4cfc-9073-646c4ea795cf","Type":"ContainerStarted","Data":"a95fe4900d4a7972d9b06d97ddd1ad431313e1ceba1216b9ad7532fac0903c80"}
Oct 29 22:11:57.635821 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:57.635796    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44" event={"ID":"674cc938-504c-4547-abb7-cc2ee413d6cf","Type":"ContainerStarted","Data":"f0ae3ab952429c68fc69b2841cf979f2f61d1493d1c65e8a4410503adf29d4f6"}
Oct 29 22:11:57.727275 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:57.724312    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2" podStartSLOduration=6.724273179 podStartE2EDuration="6.724273179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:57.686664132 +0000 UTC m=+1538.759621109" watchObservedRunningTime="2026-10-29 22:11:57.724273179 +0000 UTC m=+1538.797230144"
Oct 29 22:11:57.979225 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:57.979184    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body=
Oct 29 22:11:57.979513 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:57.979462    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s" podUID="b4cda533-309f-46e1-8cad-7239630ac19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused"
Oct 29 22:11:58.171092 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:58.171053    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 29 22:11:58.175368 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:58.175326    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw" podUID="e8c4fd64-8818-475c-b3d0-579054bcd76a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 29 22:11:58.652443 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:58.652395    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44" event={"ID":"674cc938-504c-4547-abb7-cc2ee413d6cf","Type":"ContainerStarted","Data":"76e8b0fde55763dad76d081e0cae0454330942ca4717ffa626a854c256925a63"}
Oct 29 22:11:58.652871 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:58.652509    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44"
Oct 29 22:11:58.708028 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:58.707975    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44" podStartSLOduration=5.016758614 podStartE2EDuration="5.707962455s" totalImagesPullingTime="691.203841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:11:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:11:58.705077946 +0000 UTC m=+1539.778034910" watchObservedRunningTime="2026-10-29 22:11:58.707962455 +0000 UTC m=+1539.780919419"
Oct 29 22:11:59.082120 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:59.081588    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:11:59.082120 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:59.081650    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57" podUID="67b4a685-bfa5-4a36-adf7-68c7591137c3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:11:59.288969 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:59.288931    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:59.290715 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:59.290691    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:11:59.319799 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:59.319768    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:59.321012 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:59.320986    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:11:59.331027 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:59.330935    4962 patch_prober.go:29] interesting pod/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:11:59.331027 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:11:59.330985    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp" podUID="5d5e71cf-5078-4917-94b8-2b531b143393" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:12:00.364952 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.364332    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:12:00.446339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.444911    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-workload-certs\") pod \"5d5e71cf-5078-4917-94b8-2b531b143393\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") "
Oct 29 22:12:00.446339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.445033    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d5e71cf-5078-4917-94b8-2b531b143393-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5e71cf-5078-4917-94b8-2b531b143393-istio-podinfo\") pod \"5d5e71cf-5078-4917-94b8-2b531b143393\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") "
Oct 29 22:12:00.446339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.445085    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-workload-socket\") pod \"5d5e71cf-5078-4917-94b8-2b531b143393\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") "
Oct 29 22:12:00.446339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.445155    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d5e71cf-5078-4917-94b8-2b531b143393-kube-api-access-zbzxf\" (UniqueName: \"kubernetes.io/projected/5d5e71cf-5078-4917-94b8-2b531b143393-kube-api-access-zbzxf\") pod \"5d5e71cf-5078-4917-94b8-2b531b143393\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") "
Oct 29 22:12:00.446339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.445403    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-istio-data\") pod \"5d5e71cf-5078-4917-94b8-2b531b143393\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") "
Oct 29 22:12:00.446339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.445554    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d5e71cf-5078-4917-94b8-2b531b143393-istio-token\" (UniqueName: \"kubernetes.io/projected/5d5e71cf-5078-4917-94b8-2b531b143393-istio-token\") pod \"5d5e71cf-5078-4917-94b8-2b531b143393\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") "
Oct 29 22:12:00.446339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.445591    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d5e71cf-5078-4917-94b8-2b531b143393-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5e71cf-5078-4917-94b8-2b531b143393-istiod-ca-cert\") pod \"5d5e71cf-5078-4917-94b8-2b531b143393\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") "
Oct 29 22:12:00.446339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.445672    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-credential-socket\") pod \"5d5e71cf-5078-4917-94b8-2b531b143393\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") "
Oct 29 22:12:00.446339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.445700    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-istio-envoy\") pod \"5d5e71cf-5078-4917-94b8-2b531b143393\" (UID: \"5d5e71cf-5078-4917-94b8-2b531b143393\") "
Oct 29 22:12:00.448358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.448068    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-workload-certs" pod "5d5e71cf-5078-4917-94b8-2b531b143393" (UID: "5d5e71cf-5078-4917-94b8-2b531b143393"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.448358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.448241    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-istio-data" pod "5d5e71cf-5078-4917-94b8-2b531b143393" (UID: "5d5e71cf-5078-4917-94b8-2b531b143393"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.448358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.448326    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-workload-socket" pod "5d5e71cf-5078-4917-94b8-2b531b143393" (UID: "5d5e71cf-5078-4917-94b8-2b531b143393"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.448635 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.448379    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-credential-socket" pod "5d5e71cf-5078-4917-94b8-2b531b143393" (UID: "5d5e71cf-5078-4917-94b8-2b531b143393"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.448635 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.448417    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d5e71cf-5078-4917-94b8-2b531b143393-istiod-ca-cert" pod "5d5e71cf-5078-4917-94b8-2b531b143393" (UID: "5d5e71cf-5078-4917-94b8-2b531b143393"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:00.454653 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.454606    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5e71cf-5078-4917-94b8-2b531b143393-kube-api-access-zbzxf" pod "5d5e71cf-5078-4917-94b8-2b531b143393" (UID: "5d5e71cf-5078-4917-94b8-2b531b143393"). InnerVolumeSpecName "kube-api-access-zbzxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:00.455689 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.455664    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d5e71cf-5078-4917-94b8-2b531b143393-istio-podinfo" pod "5d5e71cf-5078-4917-94b8-2b531b143393" (UID: "5d5e71cf-5078-4917-94b8-2b531b143393"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:12:00.458795 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.458720    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5e71cf-5078-4917-94b8-2b531b143393-istio-token" pod "5d5e71cf-5078-4917-94b8-2b531b143393" (UID: "5d5e71cf-5078-4917-94b8-2b531b143393"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:00.460820 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.460723    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-istio-envoy" pod "5d5e71cf-5078-4917-94b8-2b531b143393" (UID: "5d5e71cf-5078-4917-94b8-2b531b143393"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.549033 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.548997    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5e71cf-5078-4917-94b8-2b531b143393-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.549033 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.549030    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5e71cf-5078-4917-94b8-2b531b143393-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.549250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.549044    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.549250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.549060    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.549250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.549074    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.549250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.549088    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5e71cf-5078-4917-94b8-2b531b143393-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.549250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.549101    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.549250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.549115    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbzxf\" (UniqueName: \"kubernetes.io/projected/5d5e71cf-5078-4917-94b8-2b531b143393-kube-api-access-zbzxf\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.549250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.549129    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5e71cf-5078-4917-94b8-2b531b143393-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.578643 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.578614    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 22:12:00.697252 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.697217    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:12:00.734229 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.734195    4962 generic.go:356] "Generic (PLEG): container finished" podID="5d5e71cf-5078-4917-94b8-2b531b143393" containerID="3c7774bee68cad1eae65e008896454a04da85c5ea27d3083c5f4af47e55854ad" exitCode=0
Oct 29 22:12:00.734411 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.734251    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"
Oct 29 22:12:00.734411 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.734256    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp" event={"ID":"5d5e71cf-5078-4917-94b8-2b531b143393","Type":"ContainerDied","Data":"3c7774bee68cad1eae65e008896454a04da85c5ea27d3083c5f4af47e55854ad"}
Oct 29 22:12:00.734411 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.734355    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp" event={"ID":"5d5e71cf-5078-4917-94b8-2b531b143393","Type":"ContainerDied","Data":"e34b36b425dc454b9b545a2b456ab70f74cf06c4d4403e877db11d5d7f56be6a"}
Oct 29 22:12:00.734411 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.734381    4962 scope.go:122] "RemoveContainer" containerID="3c7774bee68cad1eae65e008896454a04da85c5ea27d3083c5f4af47e55854ad"
Oct 29 22:12:00.736350 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.736136    4962 generic.go:356] "Generic (PLEG): container finished" podID="84e0c5a1-5acf-4a80-8448-5aec9f52b0b8" containerID="b8938ecc7272a4a12de802fbe8b51101e19c7cf48dc91072654e0d3b6a0642ca" exitCode=0
Oct 29 22:12:00.736350 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.736164    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf" event={"ID":"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8","Type":"ContainerDied","Data":"b8938ecc7272a4a12de802fbe8b51101e19c7cf48dc91072654e0d3b6a0642ca"}
Oct 29 22:12:00.736350 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.736185    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"
Oct 29 22:12:00.736350 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.736205    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf" event={"ID":"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8","Type":"ContainerDied","Data":"2d6dad9233bd211c3c0e6e6725212a930fb91ec67a96ecc6e207ebdc439aaa89"}
Oct 29 22:12:00.738641 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.738618    4962 generic.go:356] "Generic (PLEG): container finished" podID="d71de0ec-1eaa-481f-8cee-cb256e330136" containerID="c7bbd6fb4c953b189e12f0cbdc8d2640a8a4666d5e467a6b82d2253dd3127a19" exitCode=0
Oct 29 22:12:00.738780 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.738666    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"
Oct 29 22:12:00.738780 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.738704    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv" event={"ID":"d71de0ec-1eaa-481f-8cee-cb256e330136","Type":"ContainerDied","Data":"c7bbd6fb4c953b189e12f0cbdc8d2640a8a4666d5e467a6b82d2253dd3127a19"}
Oct 29 22:12:00.738780 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.738757    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv" event={"ID":"d71de0ec-1eaa-481f-8cee-cb256e330136","Type":"ContainerDied","Data":"1dfb81cf5e3c8a2f5616e698d0d525feed533a64d9ffe0c4fa75273f732e2474"}
Oct 29 22:12:00.740794 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.740769    4962 generic.go:356] "Generic (PLEG): container finished" podID="67b4a685-bfa5-4a36-adf7-68c7591137c3" containerID="733129bde7f2b2532aaf9c3791293ff0e39979c9dafb1975650ec8f7f4bdd8fd" exitCode=0
Oct 29 22:12:00.740908 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.740837    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57" event={"ID":"67b4a685-bfa5-4a36-adf7-68c7591137c3","Type":"ContainerDied","Data":"733129bde7f2b2532aaf9c3791293ff0e39979c9dafb1975650ec8f7f4bdd8fd"}
Oct 29 22:12:00.750238 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.750213    4962 scope.go:122] "RemoveContainer" containerID="3c7774bee68cad1eae65e008896454a04da85c5ea27d3083c5f4af47e55854ad"
Oct 29 22:12:00.750640 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:12:00.750610    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c7774bee68cad1eae65e008896454a04da85c5ea27d3083c5f4af47e55854ad\": container with ID starting with 3c7774bee68cad1eae65e008896454a04da85c5ea27d3083c5f4af47e55854ad not found: ID does not exist" containerID="3c7774bee68cad1eae65e008896454a04da85c5ea27d3083c5f4af47e55854ad"
Oct 29 22:12:00.750752 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.750649    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c7774bee68cad1eae65e008896454a04da85c5ea27d3083c5f4af47e55854ad"} err="failed to get container status \"3c7774bee68cad1eae65e008896454a04da85c5ea27d3083c5f4af47e55854ad\": rpc error: code = NotFound desc = could not find container \"3c7774bee68cad1eae65e008896454a04da85c5ea27d3083c5f4af47e55854ad\": container with ID starting with 3c7774bee68cad1eae65e008896454a04da85c5ea27d3083c5f4af47e55854ad not found: ID does not exist"
Oct 29 22:12:00.750752 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.750673    4962 scope.go:122] "RemoveContainer" containerID="b8938ecc7272a4a12de802fbe8b51101e19c7cf48dc91072654e0d3b6a0642ca"
Oct 29 22:12:00.755397 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.753165    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-podinfo\") pod \"d71de0ec-1eaa-481f-8cee-cb256e330136\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") "
Oct 29 22:12:00.755397 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.753217    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-workload-certs\") pod \"d71de0ec-1eaa-481f-8cee-cb256e330136\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") "
Oct 29 22:12:00.755397 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.753343    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-workload-socket\") pod \"d71de0ec-1eaa-481f-8cee-cb256e330136\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") "
Oct 29 22:12:00.755397 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.753417    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-envoy\") pod \"d71de0ec-1eaa-481f-8cee-cb256e330136\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") "
Oct 29 22:12:00.755397 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.753470    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d71de0ec-1eaa-481f-8cee-cb256e330136-kube-api-access-ws6dp\" (UniqueName: \"kubernetes.io/projected/d71de0ec-1eaa-481f-8cee-cb256e330136-kube-api-access-ws6dp\") pod \"d71de0ec-1eaa-481f-8cee-cb256e330136\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") "
Oct 29 22:12:00.755397 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.753571    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-credential-socket\") pod \"d71de0ec-1eaa-481f-8cee-cb256e330136\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") "
Oct 29 22:12:00.755397 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.753617    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-token\" (UniqueName: \"kubernetes.io/projected/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-token\") pod \"d71de0ec-1eaa-481f-8cee-cb256e330136\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") "
Oct 29 22:12:00.755397 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.753657    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-data\") pod \"d71de0ec-1eaa-481f-8cee-cb256e330136\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") "
Oct 29 22:12:00.755397 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.753742    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d71de0ec-1eaa-481f-8cee-cb256e330136-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d71de0ec-1eaa-481f-8cee-cb256e330136-istiod-ca-cert\") pod \"d71de0ec-1eaa-481f-8cee-cb256e330136\" (UID: \"d71de0ec-1eaa-481f-8cee-cb256e330136\") "
Oct 29 22:12:00.755397 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.753818    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-workload-socket" pod "d71de0ec-1eaa-481f-8cee-cb256e330136" (UID: "d71de0ec-1eaa-481f-8cee-cb256e330136"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.755397 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.754057    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-workload-certs" pod "d71de0ec-1eaa-481f-8cee-cb256e330136" (UID: "d71de0ec-1eaa-481f-8cee-cb256e330136"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.755397 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.754260    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-credential-socket" pod "d71de0ec-1eaa-481f-8cee-cb256e330136" (UID: "d71de0ec-1eaa-481f-8cee-cb256e330136"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.755397 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.754793    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.755397 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.754816    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.755397 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.754829    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.756051 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.755958    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71de0ec-1eaa-481f-8cee-cb256e330136-istiod-ca-cert" pod "d71de0ec-1eaa-481f-8cee-cb256e330136" (UID: "d71de0ec-1eaa-481f-8cee-cb256e330136"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:00.756319 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.756278    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-data" pod "d71de0ec-1eaa-481f-8cee-cb256e330136" (UID: "d71de0ec-1eaa-481f-8cee-cb256e330136"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.760840 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.760806    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71de0ec-1eaa-481f-8cee-cb256e330136-kube-api-access-ws6dp" pod "d71de0ec-1eaa-481f-8cee-cb256e330136" (UID: "d71de0ec-1eaa-481f-8cee-cb256e330136"). InnerVolumeSpecName "kube-api-access-ws6dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:00.767718 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.764444    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:12:00.767718 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.764675    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-podinfo" pod "d71de0ec-1eaa-481f-8cee-cb256e330136" (UID: "d71de0ec-1eaa-481f-8cee-cb256e330136"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:12:00.771509 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.769053    4962 scope.go:122] "RemoveContainer" containerID="b8938ecc7272a4a12de802fbe8b51101e19c7cf48dc91072654e0d3b6a0642ca"
Oct 29 22:12:00.771509 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:12:00.769448    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8938ecc7272a4a12de802fbe8b51101e19c7cf48dc91072654e0d3b6a0642ca\": container with ID starting with b8938ecc7272a4a12de802fbe8b51101e19c7cf48dc91072654e0d3b6a0642ca not found: ID does not exist" containerID="b8938ecc7272a4a12de802fbe8b51101e19c7cf48dc91072654e0d3b6a0642ca"
Oct 29 22:12:00.771509 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.769482    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8938ecc7272a4a12de802fbe8b51101e19c7cf48dc91072654e0d3b6a0642ca"} err="failed to get container status \"b8938ecc7272a4a12de802fbe8b51101e19c7cf48dc91072654e0d3b6a0642ca\": rpc error: code = NotFound desc = could not find container \"b8938ecc7272a4a12de802fbe8b51101e19c7cf48dc91072654e0d3b6a0642ca\": container with ID starting with b8938ecc7272a4a12de802fbe8b51101e19c7cf48dc91072654e0d3b6a0642ca not found: ID does not exist"
Oct 29 22:12:00.771509 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.769505    4962 scope.go:122] "RemoveContainer" containerID="c7bbd6fb4c953b189e12f0cbdc8d2640a8a4666d5e467a6b82d2253dd3127a19"
Oct 29 22:12:00.777064 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.777030    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-token" pod "d71de0ec-1eaa-481f-8cee-cb256e330136" (UID: "d71de0ec-1eaa-481f-8cee-cb256e330136"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:00.783249 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.783226    4962 scope.go:122] "RemoveContainer" containerID="7d7c79aa202bea72b52ec17fff7e032f5c0c30df32d9e7ee9b73879951fd5010"
Oct 29 22:12:00.788448 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.788414    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-envoy" pod "d71de0ec-1eaa-481f-8cee-cb256e330136" (UID: "d71de0ec-1eaa-481f-8cee-cb256e330136"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.795446 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.793965    4962 scope.go:122] "RemoveContainer" containerID="c7bbd6fb4c953b189e12f0cbdc8d2640a8a4666d5e467a6b82d2253dd3127a19"
Oct 29 22:12:00.795446 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:12:00.794387    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7bbd6fb4c953b189e12f0cbdc8d2640a8a4666d5e467a6b82d2253dd3127a19\": container with ID starting with c7bbd6fb4c953b189e12f0cbdc8d2640a8a4666d5e467a6b82d2253dd3127a19 not found: ID does not exist" containerID="c7bbd6fb4c953b189e12f0cbdc8d2640a8a4666d5e467a6b82d2253dd3127a19"
Oct 29 22:12:00.795446 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.794452    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7bbd6fb4c953b189e12f0cbdc8d2640a8a4666d5e467a6b82d2253dd3127a19"} err="failed to get container status \"c7bbd6fb4c953b189e12f0cbdc8d2640a8a4666d5e467a6b82d2253dd3127a19\": rpc error: code = NotFound desc = could not find container \"c7bbd6fb4c953b189e12f0cbdc8d2640a8a4666d5e467a6b82d2253dd3127a19\": container with ID starting with c7bbd6fb4c953b189e12f0cbdc8d2640a8a4666d5e467a6b82d2253dd3127a19 not found: ID does not exist"
Oct 29 22:12:00.795446 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.794509    4962 scope.go:122] "RemoveContainer" containerID="7d7c79aa202bea72b52ec17fff7e032f5c0c30df32d9e7ee9b73879951fd5010"
Oct 29 22:12:00.795446 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:12:00.794859    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d7c79aa202bea72b52ec17fff7e032f5c0c30df32d9e7ee9b73879951fd5010\": container with ID starting with 7d7c79aa202bea72b52ec17fff7e032f5c0c30df32d9e7ee9b73879951fd5010 not found: ID does not exist" containerID="7d7c79aa202bea72b52ec17fff7e032f5c0c30df32d9e7ee9b73879951fd5010"
Oct 29 22:12:00.795446 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.794894    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d7c79aa202bea72b52ec17fff7e032f5c0c30df32d9e7ee9b73879951fd5010"} err="failed to get container status \"7d7c79aa202bea72b52ec17fff7e032f5c0c30df32d9e7ee9b73879951fd5010\": rpc error: code = NotFound desc = could not find container \"7d7c79aa202bea72b52ec17fff7e032f5c0c30df32d9e7ee9b73879951fd5010\": container with ID starting with 7d7c79aa202bea72b52ec17fff7e032f5c0c30df32d9e7ee9b73879951fd5010 not found: ID does not exist"
Oct 29 22:12:00.863664 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.863622    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-token\" (UniqueName: \"kubernetes.io/projected/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-token\") pod \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") "
Oct 29 22:12:00.863848 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.863733    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-data\") pod \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") "
Oct 29 22:12:00.863916 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.863877    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-workload-socket\") pod \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") "
Oct 29 22:12:00.863979 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.863918    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-envoy\") pod \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") "
Oct 29 22:12:00.864042 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.864006    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-data\") pod \"67b4a685-bfa5-4a36-adf7-68c7591137c3\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") "
Oct 29 22:12:00.864109 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.864089    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-credential-socket\") pod \"67b4a685-bfa5-4a36-adf7-68c7591137c3\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") "
Oct 29 22:12:00.864166 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.864139    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67b4a685-bfa5-4a36-adf7-68c7591137c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b4a685-bfa5-4a36-adf7-68c7591137c3-istiod-ca-cert\") pod \"67b4a685-bfa5-4a36-adf7-68c7591137c3\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") "
Oct 29 22:12:00.864229 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.864174    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-workload-certs\") pod \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") "
Oct 29 22:12:00.864282 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.864253    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-token\" (UniqueName: \"kubernetes.io/projected/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-token\") pod \"67b4a685-bfa5-4a36-adf7-68c7591137c3\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") "
Oct 29 22:12:00.864364 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.864302    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istiod-ca-cert\") pod \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") "
Oct 29 22:12:00.864364 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.864345    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-envoy\") pod \"67b4a685-bfa5-4a36-adf7-68c7591137c3\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") "
Oct 29 22:12:00.864537 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.864517    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-podinfo\") pod \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") "
Oct 29 22:12:00.864608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.864577    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-workload-socket\") pod \"67b4a685-bfa5-4a36-adf7-68c7591137c3\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") "
Oct 29 22:12:00.864687 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.864669    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-podinfo\") pod \"67b4a685-bfa5-4a36-adf7-68c7591137c3\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") "
Oct 29 22:12:00.864750 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.864715    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-workload-certs\") pod \"67b4a685-bfa5-4a36-adf7-68c7591137c3\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") "
Oct 29 22:12:00.864804 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.864751    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-kube-api-access-q8gx9\" (UniqueName: \"kubernetes.io/projected/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-kube-api-access-q8gx9\") pod \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") "
Oct 29 22:12:00.864868 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.864818    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-credential-socket\") pod \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\" (UID: \"84e0c5a1-5acf-4a80-8448-5aec9f52b0b8\") "
Oct 29 22:12:00.864922 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.864890    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b4a685-bfa5-4a36-adf7-68c7591137c3-kube-api-access-j9xj8\" (UniqueName: \"kubernetes.io/projected/67b4a685-bfa5-4a36-adf7-68c7591137c3-kube-api-access-j9xj8\") pod \"67b4a685-bfa5-4a36-adf7-68c7591137c3\" (UID: \"67b4a685-bfa5-4a36-adf7-68c7591137c3\") "
Oct 29 22:12:00.865741 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.865703    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-workload-socket" pod "67b4a685-bfa5-4a36-adf7-68c7591137c3" (UID: "67b4a685-bfa5-4a36-adf7-68c7591137c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.866578 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.866554    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.866708 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.866694    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d71de0ec-1eaa-481f-8cee-cb256e330136-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.866819 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.866805    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.866953 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.866937    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.867058 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.867044    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.867161 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.867147    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws6dp\" (UniqueName: \"kubernetes.io/projected/d71de0ec-1eaa-481f-8cee-cb256e330136-kube-api-access-ws6dp\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.867248 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.867236    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d71de0ec-1eaa-481f-8cee-cb256e330136-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.867740 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.867718    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istiod-ca-cert" pod "84e0c5a1-5acf-4a80-8448-5aec9f52b0b8" (UID: "84e0c5a1-5acf-4a80-8448-5aec9f52b0b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:00.868684 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.868652    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b4a685-bfa5-4a36-adf7-68c7591137c3-kube-api-access-j9xj8" pod "67b4a685-bfa5-4a36-adf7-68c7591137c3" (UID: "67b4a685-bfa5-4a36-adf7-68c7591137c3"). InnerVolumeSpecName "kube-api-access-j9xj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:00.871139 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.871076    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-podinfo" pod "67b4a685-bfa5-4a36-adf7-68c7591137c3" (UID: "67b4a685-bfa5-4a36-adf7-68c7591137c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:12:00.871346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.871278    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-workload-certs" pod "67b4a685-bfa5-4a36-adf7-68c7591137c3" (UID: "67b4a685-bfa5-4a36-adf7-68c7591137c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.871644 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.871620    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-credential-socket" pod "84e0c5a1-5acf-4a80-8448-5aec9f52b0b8" (UID: "84e0c5a1-5acf-4a80-8448-5aec9f52b0b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.871909 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.871875    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-data" pod "84e0c5a1-5acf-4a80-8448-5aec9f52b0b8" (UID: "84e0c5a1-5acf-4a80-8448-5aec9f52b0b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.873674 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.873642    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-envoy" pod "84e0c5a1-5acf-4a80-8448-5aec9f52b0b8" (UID: "84e0c5a1-5acf-4a80-8448-5aec9f52b0b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.873887 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.873864    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-workload-socket" pod "84e0c5a1-5acf-4a80-8448-5aec9f52b0b8" (UID: "84e0c5a1-5acf-4a80-8448-5aec9f52b0b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.874089 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.874056    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-credential-socket" pod "67b4a685-bfa5-4a36-adf7-68c7591137c3" (UID: "67b4a685-bfa5-4a36-adf7-68c7591137c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.874381 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.874280    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-data" pod "67b4a685-bfa5-4a36-adf7-68c7591137c3" (UID: "67b4a685-bfa5-4a36-adf7-68c7591137c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.874575 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.874557    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b4a685-bfa5-4a36-adf7-68c7591137c3-istiod-ca-cert" pod "67b4a685-bfa5-4a36-adf7-68c7591137c3" (UID: "67b4a685-bfa5-4a36-adf7-68c7591137c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:00.874739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.874724    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-workload-certs" pod "84e0c5a1-5acf-4a80-8448-5aec9f52b0b8" (UID: "84e0c5a1-5acf-4a80-8448-5aec9f52b0b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.874929 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.874897    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-podinfo" pod "84e0c5a1-5acf-4a80-8448-5aec9f52b0b8" (UID: "84e0c5a1-5acf-4a80-8448-5aec9f52b0b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:12:00.875008 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.874968    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-envoy" pod "67b4a685-bfa5-4a36-adf7-68c7591137c3" (UID: "67b4a685-bfa5-4a36-adf7-68c7591137c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:00.876790 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.876769    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-kube-api-access-q8gx9" pod "84e0c5a1-5acf-4a80-8448-5aec9f52b0b8" (UID: "84e0c5a1-5acf-4a80-8448-5aec9f52b0b8"). InnerVolumeSpecName "kube-api-access-q8gx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:00.880612 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.880583    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-token" pod "84e0c5a1-5acf-4a80-8448-5aec9f52b0b8" (UID: "84e0c5a1-5acf-4a80-8448-5aec9f52b0b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:00.881680 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.881642    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-token" pod "67b4a685-bfa5-4a36-adf7-68c7591137c3" (UID: "67b4a685-bfa5-4a36-adf7-68c7591137c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:00.888373 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.888344    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"]
Oct 29 22:12:00.894300 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.894265    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-jctdp"]
Oct 29 22:12:00.968448 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.968423    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.968448 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.968445    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.968448 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.968455    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.968644 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.968463    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.968644 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.968473    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.968644 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.968483    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.968644 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.968491    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8gx9\" (UniqueName: \"kubernetes.io/projected/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-kube-api-access-q8gx9\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.968644 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.968502    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.968644 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.968511    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9xj8\" (UniqueName: \"kubernetes.io/projected/67b4a685-bfa5-4a36-adf7-68c7591137c3-kube-api-access-j9xj8\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.968644 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.968519    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.968644 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.968527    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.968644 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.968534    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.968644 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.968542    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.968644 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.968549    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.968644 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.968556    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b4a685-bfa5-4a36-adf7-68c7591137c3-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.968644 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.968564    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b4a685-bfa5-4a36-adf7-68c7591137c3-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:00.968644 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:00.968572    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:01.062804 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:01.062775    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d5e71cf-5078-4917-94b8-2b531b143393" path="/var/lib/kubelet/pods/5d5e71cf-5078-4917-94b8-2b531b143393/volumes"
Oct 29 22:12:01.092555 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:01.092393    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"]
Oct 29 22:12:01.098411 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:01.098388    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-fjzzf"]
Oct 29 22:12:01.130659 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:01.130590    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"]
Oct 29 22:12:01.136599 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:01.136574    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-vkvvv"]
Oct 29 22:12:01.798813 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:01.798786    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"
Oct 29 22:12:01.799278 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:01.798789    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57" event={"ID":"67b4a685-bfa5-4a36-adf7-68c7591137c3","Type":"ContainerDied","Data":"266454f7998a7b81c9eb10f1a66075ae36f3840edda628f4cc8146cfa6d59be3"}
Oct 29 22:12:01.799278 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:01.798926    4962 scope.go:122] "RemoveContainer" containerID="733129bde7f2b2532aaf9c3791293ff0e39979c9dafb1975650ec8f7f4bdd8fd"
Oct 29 22:12:01.851604 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:01.851568    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"]
Oct 29 22:12:01.856620 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:01.856596    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a9def7-openshift-default-7fd85dc557-njx57"]
Oct 29 22:12:02.479280 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:02.476960    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": read tcp 10.132.0.2:40576->10.132.0.150:15021: read: connection reset by peer" start-of-body=
Oct 29 22:12:02.479280 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:02.477035    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm" podUID="faea6cb2-afaf-49a8-81ab-bb9c3f699e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": read tcp 10.132.0.2:40576->10.132.0.150:15021: read: connection reset by peer"
Oct 29 22:12:02.810881 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:02.810796    4962 generic.go:356] "Generic (PLEG): container finished" podID="faea6cb2-afaf-49a8-81ab-bb9c3f699e35" containerID="44c462ca217edfa89d757be4543e0b8ba0dc40798afa07384667979bc9d4b870" exitCode=0
Oct 29 22:12:02.811345 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:02.810889    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm" event={"ID":"faea6cb2-afaf-49a8-81ab-bb9c3f699e35","Type":"ContainerDied","Data":"44c462ca217edfa89d757be4543e0b8ba0dc40798afa07384667979bc9d4b870"}
Oct 29 22:12:02.811467 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:02.811450    4962 scope.go:122] "RemoveContainer" containerID="44c462ca217edfa89d757be4543e0b8ba0dc40798afa07384667979bc9d4b870"
Oct 29 22:12:02.813401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:02.813373    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:12:02.813498 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:02.813440    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:12:02.813498 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:02.813475    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:12:03.062200 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:03.062109    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67b4a685-bfa5-4a36-adf7-68c7591137c3" path="/var/lib/kubelet/pods/67b4a685-bfa5-4a36-adf7-68c7591137c3/volumes"
Oct 29 22:12:03.062927 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:03.062896    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e0c5a1-5acf-4a80-8448-5aec9f52b0b8" path="/var/lib/kubelet/pods/84e0c5a1-5acf-4a80-8448-5aec9f52b0b8/volumes"
Oct 29 22:12:03.063764 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:03.063744    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d71de0ec-1eaa-481f-8cee-cb256e330136" path="/var/lib/kubelet/pods/d71de0ec-1eaa-481f-8cee-cb256e330136/volumes"
Oct 29 22:12:03.823113 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:03.823079    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm" event={"ID":"faea6cb2-afaf-49a8-81ab-bb9c3f699e35","Type":"ContainerStarted","Data":"81135eb7acd2c9858e998d0b27e6fcbdaaa553970f782deec510669bf5b1f318"}
Oct 29 22:12:04.096895 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:04.096797    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:12:04.097082 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:04.096871    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 29 22:12:04.097082 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:04.097008    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm" podUID="faea6cb2-afaf-49a8-81ab-bb9c3f699e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 29 22:12:04.731656 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:04.731624    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44"
Oct 29 22:12:05.096733 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:05.096644    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 29 22:12:05.097405 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:05.097362    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm" podUID="faea6cb2-afaf-49a8-81ab-bb9c3f699e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 29 22:12:06.091311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:06.081587    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 29 22:12:06.091311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:06.081645    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm" podUID="faea6cb2-afaf-49a8-81ab-bb9c3f699e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 29 22:12:07.090338 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:07.083328    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 29 22:12:07.090338 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:07.083410    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm" podUID="faea6cb2-afaf-49a8-81ab-bb9c3f699e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 29 22:12:08.087643 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:08.087606    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 29 22:12:08.087927 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:08.087901    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm" podUID="faea6cb2-afaf-49a8-81ab-bb9c3f699e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 29 22:12:08.680717 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:08.680639    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2"
Oct 29 22:12:09.106020 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.105364    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body=
Oct 29 22:12:09.106020 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.105444    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm" podUID="faea6cb2-afaf-49a8-81ab-bb9c3f699e35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused"
Oct 29 22:12:09.471164 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.471127    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"]
Oct 29 22:12:09.472024 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.472001    4962 cpu_manager.go:414] "containerMap: removing container" podUID="d71de0ec-1eaa-481f-8cee-cb256e330136" containerName="istio-proxy"
Oct 29 22:12:09.472024 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.472023    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71de0ec-1eaa-481f-8cee-cb256e330136" containerName="istio-proxy"
Oct 29 22:12:09.472227 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.472064    4962 cpu_manager.go:414] "containerMap: removing container" podUID="d71de0ec-1eaa-481f-8cee-cb256e330136" containerName="istio-proxy"
Oct 29 22:12:09.472227 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.472073    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71de0ec-1eaa-481f-8cee-cb256e330136" containerName="istio-proxy"
Oct 29 22:12:09.472227 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.472088    4962 cpu_manager.go:414] "containerMap: removing container" podUID="67b4a685-bfa5-4a36-adf7-68c7591137c3" containerName="istio-proxy"
Oct 29 22:12:09.472227 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.472095    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b4a685-bfa5-4a36-adf7-68c7591137c3" containerName="istio-proxy"
Oct 29 22:12:09.472227 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.472125    4962 cpu_manager.go:414] "containerMap: removing container" podUID="84e0c5a1-5acf-4a80-8448-5aec9f52b0b8" containerName="istio-proxy"
Oct 29 22:12:09.472227 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.472132    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e0c5a1-5acf-4a80-8448-5aec9f52b0b8" containerName="istio-proxy"
Oct 29 22:12:09.472227 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.472148    4962 cpu_manager.go:414] "containerMap: removing container" podUID="5d5e71cf-5078-4917-94b8-2b531b143393" containerName="istio-proxy"
Oct 29 22:12:09.472227 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.472155    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d5e71cf-5078-4917-94b8-2b531b143393" containerName="istio-proxy"
Oct 29 22:12:09.472675 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.472419    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71de0ec-1eaa-481f-8cee-cb256e330136" containerName="istio-proxy"
Oct 29 22:12:09.472675 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.472444    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71de0ec-1eaa-481f-8cee-cb256e330136" containerName="istio-proxy"
Oct 29 22:12:09.472675 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.472456    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b4a685-bfa5-4a36-adf7-68c7591137c3" containerName="istio-proxy"
Oct 29 22:12:09.472675 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.472469    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e0c5a1-5acf-4a80-8448-5aec9f52b0b8" containerName="istio-proxy"
Oct 29 22:12:09.472675 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.472479    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d5e71cf-5078-4917-94b8-2b531b143393" containerName="istio-proxy"
Oct 29 22:12:09.477988 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.477965    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.492223 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.492200    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-48c52b-openshift-default-dockercfg-zsvgm\""
Oct 29 22:12:09.509805 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.509773    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"]
Oct 29 22:12:09.531748 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.531713    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.531923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.531896    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.532064 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.532045    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.532120 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.532103    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lj5s\" (UniqueName: \"kubernetes.io/projected/13d62170-e1c5-4478-886e-1b7b7f20d9e0-kube-api-access-8lj5s\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.532181 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.532167    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.532242 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.532227    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.532314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.532271    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.532478 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.532456    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.532528 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.532510    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.633656 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.633624    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lj5s\" (UniqueName: \"kubernetes.io/projected/13d62170-e1c5-4478-886e-1b7b7f20d9e0-kube-api-access-8lj5s\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.633824 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.633677    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.633824 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.633734    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.633824 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.633767    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.633824 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.633808    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.634046 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.633841    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.634046 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.633966    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.634162 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.634063    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.634162 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.634128    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.634304 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.634166    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.634388 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.634352    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.634388 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.634375    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.634517 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.634469    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.635175 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.635147    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.636015 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.635993    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.636420 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.636403    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.656045 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.656025    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lj5s\" (UniqueName: \"kubernetes.io/projected/13d62170-e1c5-4478-886e-1b7b7f20d9e0-kube-api-access-8lj5s\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.656145 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.656057    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:09.722461 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:12:09.722381    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:12:09.795516 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:09.795487    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:10.085145 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:10.085072    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:12:10.085557 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:10.085542    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:12:10.086093 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:10.086073    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:12:10.125828 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:10.125803    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"]
Oct 29 22:12:10.126316 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:12:10.126269    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d62170_e1c5_4478_886e_1b7b7f20d9e0.slice/crio-b16e3bbbb558ad404267506f499c472f0f740a4ac15573ad556561ccef20abaf WatchSource:0}: Error finding container b16e3bbbb558ad404267506f499c472f0f740a4ac15573ad556561ccef20abaf: Status 404 returned error can't find the container with id b16e3bbbb558ad404267506f499c472f0f740a4ac15573ad556561ccef20abaf
Oct 29 22:12:10.129084 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:10.129053    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:12:10.129165 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:10.129116    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:12:10.129165 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:10.129144    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:12:10.928958 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:10.928922    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p" event={"ID":"13d62170-e1c5-4478-886e-1b7b7f20d9e0","Type":"ContainerStarted","Data":"8c351b4f6ebf661af34efaf64ed4ae5a99c86b78b241f60863c7c73b89a2a906"}
Oct 29 22:12:10.929574 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:10.928972    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p" event={"ID":"13d62170-e1c5-4478-886e-1b7b7f20d9e0","Type":"ContainerStarted","Data":"b16e3bbbb558ad404267506f499c472f0f740a4ac15573ad556561ccef20abaf"}
Oct 29 22:12:10.981064 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:10.976079    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p" podStartSLOduration=1.9760610760000001 podStartE2EDuration="1.976061076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:10.974751345 +0000 UTC m=+1552.047708311" watchObservedRunningTime="2026-10-29 22:12:10.976061076 +0000 UTC m=+1552.049018041"
Oct 29 22:12:11.797318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:11.796848    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:11.798077 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:11.798043    4962 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 29 22:12:11.798221 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:11.798109    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p" podUID="13d62170-e1c5-4478-886e-1b7b7f20d9e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 29 22:12:12.796632 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:12.796597    4962 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 29 22:12:12.797221 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:12.797188    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p" podUID="13d62170-e1c5-4478-886e-1b7b7f20d9e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 29 22:12:13.796675 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:13.796638    4962 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 29 22:12:13.797238 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:13.797203    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p" podUID="13d62170-e1c5-4478-886e-1b7b7f20d9e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 29 22:12:14.601440 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.601392    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"]
Oct 29 22:12:14.611665 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.611641    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.626415 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.626390    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-b80d5b-openshift-default-dockercfg-4wczk\""
Oct 29 22:12:14.643223 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.643195    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"]
Oct 29 22:12:14.697608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.697565    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baea6f61-3440-48ed-868f-7e2bbf77df97-istiod-ca-cert\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.697939 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.697914    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-credential-socket\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.698105 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.698085    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-podinfo\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.698268 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.698248    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-token\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.698469 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.698439    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-envoy\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.698632 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.698594    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cwff\" (UniqueName: \"kubernetes.io/projected/baea6f61-3440-48ed-868f-7e2bbf77df97-kube-api-access-8cwff\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.698700 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.698651    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-data\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.698914 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.698892    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-workload-certs\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.698987 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.698939    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-workload-socket\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.797519 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.797481    4962 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 29 22:12:14.798022 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.797541    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p" podUID="13d62170-e1c5-4478-886e-1b7b7f20d9e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 29 22:12:14.800445 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.800416    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-credential-socket\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.800591 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.800458    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-podinfo\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.800591 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.800502    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-token\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.800591 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.800531    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-envoy\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.800774 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.800622    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cwff\" (UniqueName: \"kubernetes.io/projected/baea6f61-3440-48ed-868f-7e2bbf77df97-kube-api-access-8cwff\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.800774 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.800657    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-data\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.800904 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.800845    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-credential-socket\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.800904 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.800891    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-workload-certs\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.801021 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.800935    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-workload-socket\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.801088 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.801037    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baea6f61-3440-48ed-868f-7e2bbf77df97-istiod-ca-cert\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.801253 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.801214    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-data\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.801572 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.801554    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-workload-socket\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.801670 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.801592    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-workload-certs\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.802034 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.801994    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baea6f61-3440-48ed-868f-7e2bbf77df97-istiod-ca-cert\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.803393 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.803372    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-envoy\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.803714 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.803696    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-podinfo\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.825009 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.824993    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-token\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.825127 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.825102    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cwff\" (UniqueName: \"kubernetes.io/projected/baea6f61-3440-48ed-868f-7e2bbf77df97-kube-api-access-8cwff\") pod \"e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") " pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:14.928369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:14.928338    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:15.324244 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:12:15.324214    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaea6f61_3440_48ed_868f_7e2bbf77df97.slice/crio-17880cbc4daed9cad8d406f0300db6e231fce55fd0d78dfe9a4ebb029b0af954 WatchSource:0}: Error finding container 17880cbc4daed9cad8d406f0300db6e231fce55fd0d78dfe9a4ebb029b0af954: Status 404 returned error can't find the container with id 17880cbc4daed9cad8d406f0300db6e231fce55fd0d78dfe9a4ebb029b0af954
Oct 29 22:12:15.324968 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:15.324941    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"]
Oct 29 22:12:15.327705 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:15.327671    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:12:15.327793 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:15.327760    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:12:15.327834 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:15.327804    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:12:15.706309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:15.705368    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc"]
Oct 29 22:12:15.713253 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:15.713225    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc"]
Oct 29 22:12:15.713427 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:15.713346    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc"
Oct 29 22:12:15.733762 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:15.732583    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-75db5b-serving-cert\""
Oct 29 22:12:15.804345 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:15.802888    4962 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Oct 29 22:12:15.804345 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:15.802954    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p" podUID="13d62170-e1c5-4478-886e-1b7b7f20d9e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Oct 29 22:12:15.820735 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:15.820683    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc\" (UID: \"f18a68f6-5a07-41c2-acbe-58a2c98ebdd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc"
Oct 29 22:12:15.820916 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:15.820754    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dtz2\" (UniqueName: \"kubernetes.io/projected/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-kube-api-access-6dtz2\") pod \"maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc\" (UID: \"f18a68f6-5a07-41c2-acbe-58a2c98ebdd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc"
Oct 29 22:12:15.820916 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:15.820843    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc\" (UID: \"f18a68f6-5a07-41c2-acbe-58a2c98ebdd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc"
Oct 29 22:12:15.821279 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:15.821053    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-metrics-certs\") pod \"maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc\" (UID: \"f18a68f6-5a07-41c2-acbe-58a2c98ebdd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc"
Oct 29 22:12:15.922743 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:15.922701    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-metrics-certs\") pod \"maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc\" (UID: \"f18a68f6-5a07-41c2-acbe-58a2c98ebdd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc"
Oct 29 22:12:15.922949 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:15.922930    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc\" (UID: \"f18a68f6-5a07-41c2-acbe-58a2c98ebdd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc"
Oct 29 22:12:15.923007 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:15.922976    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dtz2\" (UniqueName: \"kubernetes.io/projected/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-kube-api-access-6dtz2\") pod \"maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc\" (UID: \"f18a68f6-5a07-41c2-acbe-58a2c98ebdd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc"
Oct 29 22:12:15.923053 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:15.923024    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc\" (UID: \"f18a68f6-5a07-41c2-acbe-58a2c98ebdd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc"
Oct 29 22:12:15.924181 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:15.924156    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc\" (UID: \"f18a68f6-5a07-41c2-acbe-58a2c98ebdd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc"
Oct 29 22:12:15.928561 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:15.928262    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc\" (UID: \"f18a68f6-5a07-41c2-acbe-58a2c98ebdd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc"
Oct 29 22:12:15.929233 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:15.929210    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-metrics-certs\") pod \"maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc\" (UID: \"f18a68f6-5a07-41c2-acbe-58a2c98ebdd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc"
Oct 29 22:12:15.947682 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:15.947652    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dtz2\" (UniqueName: \"kubernetes.io/projected/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-kube-api-access-6dtz2\") pod \"maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc\" (UID: \"f18a68f6-5a07-41c2-acbe-58a2c98ebdd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc"
Oct 29 22:12:16.047310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:16.042934    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc"
Oct 29 22:12:16.132312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:16.112122    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk" event={"ID":"baea6f61-3440-48ed-868f-7e2bbf77df97","Type":"ContainerStarted","Data":"f8c016cecf7de09ad241211316a0c3096fea936e87ec6be4422513fd7e300e6e"}
Oct 29 22:12:16.132312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:16.112205    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk" event={"ID":"baea6f61-3440-48ed-868f-7e2bbf77df97","Type":"ContainerStarted","Data":"17880cbc4daed9cad8d406f0300db6e231fce55fd0d78dfe9a4ebb029b0af954"}
Oct 29 22:12:16.943477 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:16.936704    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:16.968785 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:16.950410    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 29 22:12:16.968785 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:16.950474    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk" podUID="baea6f61-3440-48ed-868f-7e2bbf77df97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 29 22:12:17.151314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:17.151233    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk" podStartSLOduration=3.151210307 podStartE2EDuration="3.151210307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:16.22411952 +0000 UTC m=+1557.297076485" watchObservedRunningTime="2026-10-29 22:12:17.151210307 +0000 UTC m=+1558.224167273"
Oct 29 22:12:17.230054 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:17.229957    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:17.230642 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:17.230618    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc"]
Oct 29 22:12:17.245663 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:17.242778    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:17.246017 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:17.245993    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:12:17.246205 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:17.246181    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc" event={"ID":"f18a68f6-5a07-41c2-acbe-58a2c98ebdd7","Type":"ContainerStarted","Data":"a9a03b9165c8e69ea5c465149b70747fc397ddf70e85e62ff0ebc4cf089ee9f6"}
Oct 29 22:12:17.931834 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:17.929508    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 29 22:12:17.931834 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:17.929554    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk" podUID="baea6f61-3440-48ed-868f-7e2bbf77df97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 29 22:12:18.231950 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:18.231865    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc" event={"ID":"f18a68f6-5a07-41c2-acbe-58a2c98ebdd7","Type":"ContainerStarted","Data":"5214eb98fa1e5c7c76d8b3310b6504e1cdbab340877b081291304e44b2a36980"}
Oct 29 22:12:18.300516 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:18.300450    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc" podStartSLOduration=2.827673403 podStartE2EDuration="3.300429994s" totalImagesPullingTime="472.756591ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:18.288015505 +0000 UTC m=+1559.360972470" watchObservedRunningTime="2026-10-29 22:12:18.300429994 +0000 UTC m=+1559.373386959"
Oct 29 22:12:18.929635 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:18.929600    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 29 22:12:18.929825 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:18.929654    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk" podUID="baea6f61-3440-48ed-868f-7e2bbf77df97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 29 22:12:19.242087 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:19.242015    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc"
Oct 29 22:12:19.934308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:19.931936    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 29 22:12:19.934308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:19.932019    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk" podUID="baea6f61-3440-48ed-868f-7e2bbf77df97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 29 22:12:20.928846 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:20.928804    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 29 22:12:20.929374 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:20.928870    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk" podUID="baea6f61-3440-48ed-868f-7e2bbf77df97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 29 22:12:21.929426 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:21.929386    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body=
Oct 29 22:12:21.929821 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:21.929451    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk" podUID="baea6f61-3440-48ed-868f-7e2bbf77df97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused"
Oct 29 22:12:22.933876 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:22.933840    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:22.934473 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:22.934437    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:22.935317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:22.935280    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:12:25.255051 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:25.255018    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc"
Oct 29 22:12:28.022880 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:28.020398    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw"]
Oct 29 22:12:28.158392 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:28.127358    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw" podUID="8ffbb691-305b-4414-9e01-81aa65c30fa5" containerName="maas-api" containerID="cri-o://0025361fd9fb5c661e6468b977161a89901cd3b8ab1c99aaada63585c53b3428" gracePeriod=30
Oct 29 22:12:28.644012 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:28.643921    4962 generic.go:356] "Generic (PLEG): container finished" podID="8ffbb691-305b-4414-9e01-81aa65c30fa5" containerID="0025361fd9fb5c661e6468b977161a89901cd3b8ab1c99aaada63585c53b3428" exitCode=0
Oct 29 22:12:28.644554 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:28.644521    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw" event={"ID":"8ffbb691-305b-4414-9e01-81aa65c30fa5","Type":"ContainerDied","Data":"0025361fd9fb5c661e6468b977161a89901cd3b8ab1c99aaada63585c53b3428"}
Oct 29 22:12:28.885654 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:28.885617    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw"
Oct 29 22:12:28.989058 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:28.989019    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ffbb691-305b-4414-9e01-81aa65c30fa5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ffbb691-305b-4414-9e01-81aa65c30fa5-metrics-certs\") pod \"8ffbb691-305b-4414-9e01-81aa65c30fa5\" (UID: \"8ffbb691-305b-4414-9e01-81aa65c30fa5\") "
Oct 29 22:12:28.989374 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:28.989350    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ffbb691-305b-4414-9e01-81aa65c30fa5-kube-api-access-w758l\" (UniqueName: \"kubernetes.io/projected/8ffbb691-305b-4414-9e01-81aa65c30fa5-kube-api-access-w758l\") pod \"8ffbb691-305b-4414-9e01-81aa65c30fa5\" (UID: \"8ffbb691-305b-4414-9e01-81aa65c30fa5\") "
Oct 29 22:12:28.989463 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:28.989392    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ffbb691-305b-4414-9e01-81aa65c30fa5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ffbb691-305b-4414-9e01-81aa65c30fa5-maas-api-tls\") pod \"8ffbb691-305b-4414-9e01-81aa65c30fa5\" (UID: \"8ffbb691-305b-4414-9e01-81aa65c30fa5\") "
Oct 29 22:12:28.989554 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:28.989536    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ffbb691-305b-4414-9e01-81aa65c30fa5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ffbb691-305b-4414-9e01-81aa65c30fa5-openshift-service-ca\") pod \"8ffbb691-305b-4414-9e01-81aa65c30fa5\" (UID: \"8ffbb691-305b-4414-9e01-81aa65c30fa5\") "
Oct 29 22:12:28.989995 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:28.989961    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ffbb691-305b-4414-9e01-81aa65c30fa5-openshift-service-ca" pod "8ffbb691-305b-4414-9e01-81aa65c30fa5" (UID: "8ffbb691-305b-4414-9e01-81aa65c30fa5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:28.990624 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:28.990594    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ffbb691-305b-4414-9e01-81aa65c30fa5-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:28.991721 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:28.991691    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ffbb691-305b-4414-9e01-81aa65c30fa5-maas-api-tls" pod "8ffbb691-305b-4414-9e01-81aa65c30fa5" (UID: "8ffbb691-305b-4414-9e01-81aa65c30fa5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:12:28.991807 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:28.991741    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ffbb691-305b-4414-9e01-81aa65c30fa5-metrics-certs" pod "8ffbb691-305b-4414-9e01-81aa65c30fa5" (UID: "8ffbb691-305b-4414-9e01-81aa65c30fa5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:12:28.991807 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:28.991782    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ffbb691-305b-4414-9e01-81aa65c30fa5-kube-api-access-w758l" pod "8ffbb691-305b-4414-9e01-81aa65c30fa5" (UID: "8ffbb691-305b-4414-9e01-81aa65c30fa5"). InnerVolumeSpecName "kube-api-access-w758l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:29.182305 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:29.168500    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w758l\" (UniqueName: \"kubernetes.io/projected/8ffbb691-305b-4414-9e01-81aa65c30fa5-kube-api-access-w758l\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:29.183104 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:29.183076    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ffbb691-305b-4414-9e01-81aa65c30fa5-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:29.183248 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:29.183232    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ffbb691-305b-4414-9e01-81aa65c30fa5-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:29.653950 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:29.653919    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw"
Oct 29 22:12:29.654121 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:29.653955    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw" event={"ID":"8ffbb691-305b-4414-9e01-81aa65c30fa5","Type":"ContainerDied","Data":"18798c951840536d1e90ae93daa940518f3b455c1ffcbb982398172cff79cec0"}
Oct 29 22:12:29.654121 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:29.654024    4962 scope.go:122] "RemoveContainer" containerID="0025361fd9fb5c661e6468b977161a89901cd3b8ab1c99aaada63585c53b3428"
Oct 29 22:12:29.705577 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:29.705535    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw"]
Oct 29 22:12:29.717103 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:29.717080    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9ae129-6f865897f6-n6lbw"]
Oct 29 22:12:31.060999 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:31.060969    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ffbb691-305b-4414-9e01-81aa65c30fa5" path="/var/lib/kubelet/pods/8ffbb691-305b-4414-9e01-81aa65c30fa5/volumes"
Oct 29 22:12:35.915111 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:35.914922    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"]
Oct 29 22:12:35.915483 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:35.915146    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h" podUID="c1d2bd1d-783b-415a-8807-8f53396ec0ce" containerName="istio-proxy" containerID="cri-o://88d8fe31202439750dfdaba647701549355ed02ad6eda9e0ed01494a4fe6e3fd" gracePeriod=30
Oct 29 22:12:35.922400 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:35.922369    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"]
Oct 29 22:12:35.922619 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:35.922586    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z" podUID="4ad0fa83-03aa-4824-ae79-c47cf352ff6a" containerName="istio-proxy" containerID="cri-o://30ffe75ec432d510ca3d1cc3232e37e3966b0a89f17ad35abb6cd42c1a605612" gracePeriod=30
Oct 29 22:12:35.928727 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:35.928701    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"]
Oct 29 22:12:35.929000 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:35.928974    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j" podUID="4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" containerName="istio-proxy" containerID="cri-o://e817669945643f3f5e7efa4e1c816763c2c14aa7338c7fca3182566de1cad3ac" gracePeriod=30
Oct 29 22:12:39.525692 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.525658    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:12:39.526163 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.525757    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z" podUID="4ad0fa83-03aa-4824-ae79-c47cf352ff6a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:12:39.572456 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.572425    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"]
Oct 29 22:12:39.573401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.573377    4962 cpu_manager.go:414] "containerMap: removing container" podUID="8ffbb691-305b-4414-9e01-81aa65c30fa5" containerName="maas-api"
Oct 29 22:12:39.573485 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.573403    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ffbb691-305b-4414-9e01-81aa65c30fa5" containerName="maas-api"
Oct 29 22:12:39.573563 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.573542    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ffbb691-305b-4414-9e01-81aa65c30fa5" containerName="maas-api"
Oct 29 22:12:39.579767 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.579749    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.606425 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.606400    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"]
Oct 29 22:12:39.629536 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.629506    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"]
Oct 29 22:12:39.630006 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.629986    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.640840 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.640803    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"]
Oct 29 22:12:39.721912 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:12:39.721880    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:12:39.724954 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.724927    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.725086 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.724966    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.725086 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.725067    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.725168 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.725101    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.725203 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.725186    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/654ebdbf-c9c1-4571-91b1-465aa5840051-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.725241 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.725213    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.725323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.725310    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.725447 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.725436    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.725493 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.725455    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.725493 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.725473    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96q6j\" (UniqueName: \"kubernetes.io/projected/654ebdbf-c9c1-4571-91b1-465aa5840051-kube-api-access-96q6j\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.725561 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.725495    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.725659 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.725640    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.725692 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.725669    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.725795 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.725771    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.725836 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.725824    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfh9g\" (UniqueName: \"kubernetes.io/projected/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-kube-api-access-lfh9g\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.725875 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.725867    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.725911 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.725883    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.725911 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.725900    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.881360 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.846638    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.892311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.882495    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.892311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.882564    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.892311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.882668    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.892311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.882711    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.892311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.882779    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.892311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.882822    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.892311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.882892    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/654ebdbf-c9c1-4571-91b1-465aa5840051-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.892311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.882928    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.892311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.883031    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.892311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.886229    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.918697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.893398    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.918697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.893455    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.918697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.893494    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96q6j\" (UniqueName: \"kubernetes.io/projected/654ebdbf-c9c1-4571-91b1-465aa5840051-kube-api-access-96q6j\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.918697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.893590    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.918697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.893650    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.918697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.893684    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.918697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.893794    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.918697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.893861    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfh9g\" (UniqueName: \"kubernetes.io/projected/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-kube-api-access-lfh9g\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.918697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.897845    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.918697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.898174    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.918697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.898515    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.918697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.898762    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-credential-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.918697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.899721    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.918697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.899991    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-workload-certs\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.918697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.901695    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-workload-socket\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.918697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.906770    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.918697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.907650    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-data\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.921264 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.920889    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/654ebdbf-c9c1-4571-91b1-465aa5840051-istiod-ca-cert\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.932306 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.930213    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.936648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.936622    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.936759 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.936622    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-podinfo\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.937969 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.937792    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-envoy\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.940943 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.940919    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfh9g\" (UniqueName: \"kubernetes.io/projected/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-kube-api-access-lfh9g\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.941919 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.941897    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96q6j\" (UniqueName: \"kubernetes.io/projected/654ebdbf-c9c1-4571-91b1-465aa5840051-kube-api-access-96q6j\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.943606 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.943584    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-token\") pod \"e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") " pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:39.952011 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.951988    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:39.968269 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.968237    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:12:39.968381 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:39.968313    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h" podUID="c1d2bd1d-783b-415a-8807-8f53396ec0ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:12:40.195393 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:40.195345    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:40.297156 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:40.297113    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:12:40.297246 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:40.297201    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:12:40.297319 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:40.297243    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:12:40.299133 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:40.299101    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"]
Oct 29 22:12:40.516924 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:40.516888    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"]
Oct 29 22:12:40.517685 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:40.517632    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:12:40.517789 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:40.517732    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:12:40.517789 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:40.517773    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:12:40.949951 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:40.949915    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg" event={"ID":"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a","Type":"ContainerStarted","Data":"601e860facd433df5ae34264a45424e57f7390f9743d15dcbdc7d7bdd7fd9014"}
Oct 29 22:12:40.951707 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:40.951680    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg" event={"ID":"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a","Type":"ContainerStarted","Data":"eb6d4320a3dcda6889c700762319ce42f8036185b3ea674afa26b925e8bc3481"}
Oct 29 22:12:40.952416 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:40.952398    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:40.954426 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:40.954405    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv" event={"ID":"654ebdbf-c9c1-4571-91b1-465aa5840051","Type":"ContainerStarted","Data":"f686bd197707f6a55bf13ee343a2b4fb3d69320293fe1ba76fb230f5b7f6159c"}
Oct 29 22:12:40.954584 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:40.954567    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv" event={"ID":"654ebdbf-c9c1-4571-91b1-465aa5840051","Type":"ContainerStarted","Data":"293dd24f88267d659167f093fc110637d1a62bcc6278eb5d876d258b84aea2d5"}
Oct 29 22:12:41.053225 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.053166    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg" podStartSLOduration=2.053146668 podStartE2EDuration="2.053146668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:41.003905344 +0000 UTC m=+1582.076862310" watchObservedRunningTime="2026-10-29 22:12:41.053146668 +0000 UTC m=+1582.126103634"
Oct 29 22:12:41.199343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.197454    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:41.200637 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.199982    4962 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 29 22:12:41.200637 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.200037    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv" podUID="654ebdbf-c9c1-4571-91b1-465aa5840051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 29 22:12:41.421579 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.421007    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:12:41.472354 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.472116    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv" podStartSLOduration=2.472094348 podStartE2EDuration="2.472094348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:12:41.055688234 +0000 UTC m=+1582.128645200" watchObservedRunningTime="2026-10-29 22:12:41.472094348 +0000 UTC m=+1582.545051313"
Oct 29 22:12:41.552317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.519320    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-workload-socket\") pod \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") "
Oct 29 22:12:41.552317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.519455    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d2bd1d-783b-415a-8807-8f53396ec0ce-kube-api-access-4jc5c\" (UniqueName: \"kubernetes.io/projected/c1d2bd1d-783b-415a-8807-8f53396ec0ce-kube-api-access-4jc5c\") pod \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") "
Oct 29 22:12:41.552317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.519549    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-token\" (UniqueName: \"kubernetes.io/projected/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-token\") pod \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") "
Oct 29 22:12:41.552317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.519590    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-credential-socket\") pod \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") "
Oct 29 22:12:41.552317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.519624    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-workload-certs\") pod \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") "
Oct 29 22:12:41.552317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.519674    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-podinfo\") pod \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") "
Oct 29 22:12:41.552317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.531986    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-data\") pod \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") "
Oct 29 22:12:41.552317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.532056    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-envoy\") pod \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") "
Oct 29 22:12:41.552317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.532131    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istiod-ca-cert\") pod \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\" (UID: \"c1d2bd1d-783b-415a-8807-8f53396ec0ce\") "
Oct 29 22:12:41.594697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.558040    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-data" pod "c1d2bd1d-783b-415a-8807-8f53396ec0ce" (UID: "c1d2bd1d-783b-415a-8807-8f53396ec0ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:41.594697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.560484    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istiod-ca-cert" pod "c1d2bd1d-783b-415a-8807-8f53396ec0ce" (UID: "c1d2bd1d-783b-415a-8807-8f53396ec0ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:41.595091 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.595034    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-workload-socket" pod "c1d2bd1d-783b-415a-8807-8f53396ec0ce" (UID: "c1d2bd1d-783b-415a-8807-8f53396ec0ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:41.595274 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.595241    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-workload-certs" pod "c1d2bd1d-783b-415a-8807-8f53396ec0ce" (UID: "c1d2bd1d-783b-415a-8807-8f53396ec0ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:41.595462 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.595430    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-credential-socket" pod "c1d2bd1d-783b-415a-8807-8f53396ec0ce" (UID: "c1d2bd1d-783b-415a-8807-8f53396ec0ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:41.602906 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.601400    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-podinfo" pod "c1d2bd1d-783b-415a-8807-8f53396ec0ce" (UID: "c1d2bd1d-783b-415a-8807-8f53396ec0ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:12:41.678311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.652074    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.678311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.652114    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.678311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.652129    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.678311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.652142    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.678311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.652154    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.678311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.652165    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.810535 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.810429    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-envoy" pod "c1d2bd1d-783b-415a-8807-8f53396ec0ce" (UID: "c1d2bd1d-783b-415a-8807-8f53396ec0ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:41.810931 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.810904    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d2bd1d-783b-415a-8807-8f53396ec0ce-kube-api-access-4jc5c" pod "c1d2bd1d-783b-415a-8807-8f53396ec0ce" (UID: "c1d2bd1d-783b-415a-8807-8f53396ec0ce"). InnerVolumeSpecName "kube-api-access-4jc5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:41.828438 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.827643    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-token" pod "c1d2bd1d-783b-415a-8807-8f53396ec0ce" (UID: "c1d2bd1d-783b-415a-8807-8f53396ec0ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:41.916309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.910435    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.916309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.910505    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d2bd1d-783b-415a-8807-8f53396ec0ce-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:41.916309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.910521    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jc5c\" (UniqueName: \"kubernetes.io/projected/c1d2bd1d-783b-415a-8807-8f53396ec0ce-kube-api-access-4jc5c\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.010354 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:41.999951    4962 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 29 22:12:42.010354 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.000016    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg" podUID="be60e8e6-3cc5-45f2-b8f0-0d55d04e490a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 29 22:12:42.244338 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.222442    4962 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 29 22:12:42.244338 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.222510    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv" podUID="654ebdbf-c9c1-4571-91b1-465aa5840051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 29 22:12:42.266271 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.264197    4962 generic.go:356] "Generic (PLEG): container finished" podID="4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" containerID="e817669945643f3f5e7efa4e1c816763c2c14aa7338c7fca3182566de1cad3ac" exitCode=0
Oct 29 22:12:42.266271 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.264356    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j" event={"ID":"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a","Type":"ContainerDied","Data":"e817669945643f3f5e7efa4e1c816763c2c14aa7338c7fca3182566de1cad3ac"}
Oct 29 22:12:42.266271 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.264419    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j" event={"ID":"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a","Type":"ContainerDied","Data":"752897f914d6e398bff79963222fb6afc0d098a0dee026aeb18855500db817de"}
Oct 29 22:12:42.266271 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.264441    4962 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="752897f914d6e398bff79963222fb6afc0d098a0dee026aeb18855500db817de"
Oct 29 22:12:42.266271 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.265206    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:12:42.269358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.267899    4962 generic.go:356] "Generic (PLEG): container finished" podID="4ad0fa83-03aa-4824-ae79-c47cf352ff6a" containerID="30ffe75ec432d510ca3d1cc3232e37e3966b0a89f17ad35abb6cd42c1a605612" exitCode=0
Oct 29 22:12:42.269358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.267975    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z" event={"ID":"4ad0fa83-03aa-4824-ae79-c47cf352ff6a","Type":"ContainerDied","Data":"30ffe75ec432d510ca3d1cc3232e37e3966b0a89f17ad35abb6cd42c1a605612"}
Oct 29 22:12:42.295310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.270712    4962 generic.go:356] "Generic (PLEG): container finished" podID="c1d2bd1d-783b-415a-8807-8f53396ec0ce" containerID="88d8fe31202439750dfdaba647701549355ed02ad6eda9e0ed01494a4fe6e3fd" exitCode=0
Oct 29 22:12:42.295310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.271896    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"
Oct 29 22:12:42.295310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.285165    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h" event={"ID":"c1d2bd1d-783b-415a-8807-8f53396ec0ce","Type":"ContainerDied","Data":"88d8fe31202439750dfdaba647701549355ed02ad6eda9e0ed01494a4fe6e3fd"}
Oct 29 22:12:42.295310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.285253    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h" event={"ID":"c1d2bd1d-783b-415a-8807-8f53396ec0ce","Type":"ContainerDied","Data":"7e91d64313600eb9dd1084f4c8b2dd124b3b18b960654cb634390d4a53f5a064"}
Oct 29 22:12:42.295310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.285312    4962 scope.go:122] "RemoveContainer" containerID="88d8fe31202439750dfdaba647701549355ed02ad6eda9e0ed01494a4fe6e3fd"
Oct 29 22:12:42.329346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.296789    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:12:42.329346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.318694    4962 scope.go:122] "RemoveContainer" containerID="88d8fe31202439750dfdaba647701549355ed02ad6eda9e0ed01494a4fe6e3fd"
Oct 29 22:12:42.329346 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:12:42.319134    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d8fe31202439750dfdaba647701549355ed02ad6eda9e0ed01494a4fe6e3fd\": container with ID starting with 88d8fe31202439750dfdaba647701549355ed02ad6eda9e0ed01494a4fe6e3fd not found: ID does not exist" containerID="88d8fe31202439750dfdaba647701549355ed02ad6eda9e0ed01494a4fe6e3fd"
Oct 29 22:12:42.329346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.319174    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d8fe31202439750dfdaba647701549355ed02ad6eda9e0ed01494a4fe6e3fd"} err="failed to get container status \"88d8fe31202439750dfdaba647701549355ed02ad6eda9e0ed01494a4fe6e3fd\": rpc error: code = NotFound desc = could not find container \"88d8fe31202439750dfdaba647701549355ed02ad6eda9e0ed01494a4fe6e3fd\": container with ID starting with 88d8fe31202439750dfdaba647701549355ed02ad6eda9e0ed01494a4fe6e3fd not found: ID does not exist"
Oct 29 22:12:42.441349 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.441304    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-envoy\") pod \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") "
Oct 29 22:12:42.441666 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.441650    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-data\") pod \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") "
Oct 29 22:12:42.442911 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.442885    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-token\" (UniqueName: \"kubernetes.io/projected/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-token\") pod \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") "
Oct 29 22:12:42.443105 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.443080    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-envoy\") pod \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") "
Oct 29 22:12:42.443316 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.443300    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-workload-certs\") pod \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") "
Oct 29 22:12:42.443423 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.443410    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-credential-socket\") pod \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") "
Oct 29 22:12:42.443551 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.443528    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istiod-ca-cert\") pod \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") "
Oct 29 22:12:42.444750 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.444730    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-workload-socket\") pod \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") "
Oct 29 22:12:42.444943 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.444919    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-podinfo\") pod \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") "
Oct 29 22:12:42.445082 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.445069    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-kube-api-access-jzrl9\" (UniqueName: \"kubernetes.io/projected/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-kube-api-access-jzrl9\") pod \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") "
Oct 29 22:12:42.445186 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.445173    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-credential-socket\") pod \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") "
Oct 29 22:12:42.445351 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.445336    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-podinfo\") pod \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") "
Oct 29 22:12:42.445536 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.445520    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-token\" (UniqueName: \"kubernetes.io/projected/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-token\") pod \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") "
Oct 29 22:12:42.445733 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.445719    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-data\") pod \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") "
Oct 29 22:12:42.445864 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.445850    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-workload-socket\") pod \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") "
Oct 29 22:12:42.446015 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.445993    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-kube-api-access-6hvrp\" (UniqueName: \"kubernetes.io/projected/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-kube-api-access-6hvrp\") pod \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") "
Oct 29 22:12:42.446150 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.446135    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istiod-ca-cert\") pod \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\" (UID: \"4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a\") "
Oct 29 22:12:42.446568 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.446545    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-workload-certs\") pod \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\" (UID: \"4ad0fa83-03aa-4824-ae79-c47cf352ff6a\") "
Oct 29 22:12:42.448642 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.442493    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-data" pod "4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" (UID: "4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.448642 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.444074    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istiod-ca-cert" pod "4ad0fa83-03aa-4824-ae79-c47cf352ff6a" (UID: "4ad0fa83-03aa-4824-ae79-c47cf352ff6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:42.448642 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.444534    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-credential-socket" pod "4ad0fa83-03aa-4824-ae79-c47cf352ff6a" (UID: "4ad0fa83-03aa-4824-ae79-c47cf352ff6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.448642 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.444643    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-workload-certs" pod "4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" (UID: "4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.448642 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.446478    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-credential-socket" pod "4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" (UID: "4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.448642 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.446650    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-workload-socket" pod "4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" (UID: "4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.448642 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.446698    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istiod-ca-cert" pod "4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" (UID: "4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:42.448642 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.446902    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-workload-socket" pod "4ad0fa83-03aa-4824-ae79-c47cf352ff6a" (UID: "4ad0fa83-03aa-4824-ae79-c47cf352ff6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.448642 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.446932    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-data" pod "4ad0fa83-03aa-4824-ae79-c47cf352ff6a" (UID: "4ad0fa83-03aa-4824-ae79-c47cf352ff6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.448642 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.447947    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-workload-certs" pod "4ad0fa83-03aa-4824-ae79-c47cf352ff6a" (UID: "4ad0fa83-03aa-4824-ae79-c47cf352ff6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.449250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.449186    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-envoy" pod "4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" (UID: "4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.449952 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.449932    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-podinfo" pod "4ad0fa83-03aa-4824-ae79-c47cf352ff6a" (UID: "4ad0fa83-03aa-4824-ae79-c47cf352ff6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:12:42.450786 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.450739    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-token" pod "4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" (UID: "4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:42.451418 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.451384    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-kube-api-access-jzrl9" pod "4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" (UID: "4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a"). InnerVolumeSpecName "kube-api-access-jzrl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:42.451865 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.451826    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-podinfo" pod "4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" (UID: "4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:12:42.452181 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.452123    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-envoy" pod "4ad0fa83-03aa-4824-ae79-c47cf352ff6a" (UID: "4ad0fa83-03aa-4824-ae79-c47cf352ff6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:12:42.452544 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.452513    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-kube-api-access-6hvrp" pod "4ad0fa83-03aa-4824-ae79-c47cf352ff6a" (UID: "4ad0fa83-03aa-4824-ae79-c47cf352ff6a"). InnerVolumeSpecName "kube-api-access-6hvrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:42.452984 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.452958    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-token" pod "4ad0fa83-03aa-4824-ae79-c47cf352ff6a" (UID: "4ad0fa83-03aa-4824-ae79-c47cf352ff6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:42.466553 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.466531    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"]
Oct 29 22:12:42.493661 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.471665    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-7lj9h"]
Oct 29 22:12:42.549381 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.549343    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzrl9\" (UniqueName: \"kubernetes.io/projected/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-kube-api-access-jzrl9\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.549381 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.549380    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.549583 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.549391    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.549583 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.549401    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.549583 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.549409    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.549583 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.549417    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.549583 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.549426    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hvrp\" (UniqueName: \"kubernetes.io/projected/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-kube-api-access-6hvrp\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.549583 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.549434    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.549583 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.549443    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.549583 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.549450    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.549583 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.549458    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.549583 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.549467    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.549583 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.549474    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.549583 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.549482    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.549583 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.549489    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.549583 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.549498    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.549583 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.549507    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.549583 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.549514    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad0fa83-03aa-4824-ae79-c47cf352ff6a-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:42.953196 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.953162    4962 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 29 22:12:42.953366 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:42.953212    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg" podUID="be60e8e6-3cc5-45f2-b8f0-0d55d04e490a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 29 22:12:43.061108 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:43.061072    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1d2bd1d-783b-415a-8807-8f53396ec0ce" path="/var/lib/kubelet/pods/c1d2bd1d-783b-415a-8807-8f53396ec0ce/volumes"
Oct 29 22:12:43.199369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:43.196679    4962 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 29 22:12:43.199369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:43.196740    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv" podUID="654ebdbf-c9c1-4571-91b1-465aa5840051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 29 22:12:43.332822 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:43.332393    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z" event={"ID":"4ad0fa83-03aa-4824-ae79-c47cf352ff6a","Type":"ContainerDied","Data":"35b73dd9c03d27eb02282cf8ac82e7cd731df144fffbe8bd1b755898ee74b55a"}
Oct 29 22:12:43.332822 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:43.332477    4962 scope.go:122] "RemoveContainer" containerID="30ffe75ec432d510ca3d1cc3232e37e3966b0a89f17ad35abb6cd42c1a605612"
Oct 29 22:12:43.333778 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:43.333753    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"
Oct 29 22:12:43.335960 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:43.335938    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"
Oct 29 22:12:43.426452 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:43.426336    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"]
Oct 29 22:12:43.432233 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:43.431482    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-4x74z"]
Oct 29 22:12:43.454828 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:43.454754    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"]
Oct 29 22:12:43.459539 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:43.459514    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-edc2d3-openshift-default-7c9467cc57-wv47j"]
Oct 29 22:12:43.974322 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:43.971134    4962 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 29 22:12:43.974322 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:43.971195    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg" podUID="be60e8e6-3cc5-45f2-b8f0-0d55d04e490a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 29 22:12:44.201313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:44.198614    4962 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 29 22:12:44.201313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:44.198673    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv" podUID="654ebdbf-c9c1-4571-91b1-465aa5840051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 29 22:12:44.953175 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:44.953144    4962 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 29 22:12:44.953383 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:44.953194    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg" podUID="be60e8e6-3cc5-45f2-b8f0-0d55d04e490a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 29 22:12:45.060719 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:45.060688    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad0fa83-03aa-4824-ae79-c47cf352ff6a" path="/var/lib/kubelet/pods/4ad0fa83-03aa-4824-ae79-c47cf352ff6a/volumes"
Oct 29 22:12:45.061243 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:45.061228    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" path="/var/lib/kubelet/pods/4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a/volumes"
Oct 29 22:12:45.197312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:45.196354    4962 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 29 22:12:45.197312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:45.196559    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv" podUID="654ebdbf-c9c1-4571-91b1-465aa5840051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 29 22:12:45.971321 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:45.958691    4962 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 29 22:12:45.971321 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:45.958769    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg" podUID="be60e8e6-3cc5-45f2-b8f0-0d55d04e490a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 29 22:12:46.200312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:46.197601    4962 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 29 22:12:46.200312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:46.197661    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv" podUID="654ebdbf-c9c1-4571-91b1-465aa5840051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 29 22:12:46.952421 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:46.952389    4962 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 29 22:12:46.952591 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:46.952440    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg" podUID="be60e8e6-3cc5-45f2-b8f0-0d55d04e490a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 29 22:12:47.220823 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:47.220731    4962 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused" start-of-body=
Oct 29 22:12:47.221471 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:47.221437    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv" podUID="654ebdbf-c9c1-4571-91b1-465aa5840051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.179:15021/healthz/ready\": dial tcp 10.132.0.179:15021: connect: connection refused"
Oct 29 22:12:48.053255 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:48.053213    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:48.056601 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:48.054384    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:48.056992 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:48.056966    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:12:48.263644 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:48.263578    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:48.275477 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:48.264493    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:48.275477 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:48.271030    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:12:48.399834 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:48.399742    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj"]
Oct 29 22:12:50.594280 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:50.594247    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2"]
Oct 29 22:12:53.594318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:53.590448    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj" podUID="60d52b51-0159-402d-8d66-3c453a0e438f" containerName="payload-pre-processing" containerID="cri-o://f67a8d7807ac3e5b57cdaf1a01e11957d06cc25f1288bf19b41214797935c5cc" gracePeriod=25
Oct 29 22:12:54.630791 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:54.620780    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj"
Oct 29 22:12:54.753073 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:54.751370    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60d52b51-0159-402d-8d66-3c453a0e438f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60d52b51-0159-402d-8d66-3c453a0e438f-plugins-config-volume\") pod \"60d52b51-0159-402d-8d66-3c453a0e438f\" (UID: \"60d52b51-0159-402d-8d66-3c453a0e438f\") "
Oct 29 22:12:54.753073 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:54.751458    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60d52b51-0159-402d-8d66-3c453a0e438f-kube-api-access-l96hh\" (UniqueName: \"kubernetes.io/projected/60d52b51-0159-402d-8d66-3c453a0e438f-kube-api-access-l96hh\") pod \"60d52b51-0159-402d-8d66-3c453a0e438f\" (UID: \"60d52b51-0159-402d-8d66-3c453a0e438f\") "
Oct 29 22:12:54.753777 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:54.753726    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60d52b51-0159-402d-8d66-3c453a0e438f-plugins-config-volume" pod "60d52b51-0159-402d-8d66-3c453a0e438f" (UID: "60d52b51-0159-402d-8d66-3c453a0e438f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:12:54.764969 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:54.764903    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d52b51-0159-402d-8d66-3c453a0e438f-kube-api-access-l96hh" pod "60d52b51-0159-402d-8d66-3c453a0e438f" (UID: "60d52b51-0159-402d-8d66-3c453a0e438f"). InnerVolumeSpecName "kube-api-access-l96hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:12:54.824156 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:54.824105    4962 generic.go:356] "Generic (PLEG): container finished" podID="60d52b51-0159-402d-8d66-3c453a0e438f" containerID="f67a8d7807ac3e5b57cdaf1a01e11957d06cc25f1288bf19b41214797935c5cc" exitCode=0
Oct 29 22:12:54.824636 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:54.824593    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj" event={"ID":"60d52b51-0159-402d-8d66-3c453a0e438f","Type":"ContainerDied","Data":"f67a8d7807ac3e5b57cdaf1a01e11957d06cc25f1288bf19b41214797935c5cc"}
Oct 29 22:12:54.824881 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:54.824835    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj" event={"ID":"60d52b51-0159-402d-8d66-3c453a0e438f","Type":"ContainerDied","Data":"ae222378238f8e33269d0e6a32344f6b5aa1e5231314ea9f93cf9ac8e60a78e8"}
Oct 29 22:12:54.825044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:54.825028    4962 scope.go:122] "RemoveContainer" containerID="f67a8d7807ac3e5b57cdaf1a01e11957d06cc25f1288bf19b41214797935c5cc"
Oct 29 22:12:54.825367 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:54.825349    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj"
Oct 29 22:12:54.928983 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:54.928945    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60d52b51-0159-402d-8d66-3c453a0e438f-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:54.928983 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:54.928986    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l96hh\" (UniqueName: \"kubernetes.io/projected/60d52b51-0159-402d-8d66-3c453a0e438f-kube-api-access-l96hh\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:12:55.014205 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:55.014159    4962 scope.go:122] "RemoveContainer" containerID="f67a8d7807ac3e5b57cdaf1a01e11957d06cc25f1288bf19b41214797935c5cc"
Oct 29 22:12:55.015022 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:12:55.014981    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67a8d7807ac3e5b57cdaf1a01e11957d06cc25f1288bf19b41214797935c5cc\": container with ID starting with f67a8d7807ac3e5b57cdaf1a01e11957d06cc25f1288bf19b41214797935c5cc not found: ID does not exist" containerID="f67a8d7807ac3e5b57cdaf1a01e11957d06cc25f1288bf19b41214797935c5cc"
Oct 29 22:12:55.015214 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:55.015184    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67a8d7807ac3e5b57cdaf1a01e11957d06cc25f1288bf19b41214797935c5cc"} err="failed to get container status \"f67a8d7807ac3e5b57cdaf1a01e11957d06cc25f1288bf19b41214797935c5cc\": rpc error: code = NotFound desc = could not find container \"f67a8d7807ac3e5b57cdaf1a01e11957d06cc25f1288bf19b41214797935c5cc\": container with ID starting with f67a8d7807ac3e5b57cdaf1a01e11957d06cc25f1288bf19b41214797935c5cc not found: ID does not exist"
Oct 29 22:12:55.602956 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:55.602793    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2" podUID="6033cb02-b74f-4cfc-9073-646c4ea795cf" containerName="payload-pre-processing" containerID="cri-o://d2399bc9194752bfb7eaab93c62ab05edc073d29cc2195c9f4163334a7b7a772" gracePeriod=25
Oct 29 22:12:55.752383 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:55.750559    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj"]
Oct 29 22:12:55.752383 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:55.750611    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-48c52b-6f648586cf-nclwj"]
Oct 29 22:12:56.261002 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:56.257742    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:12:56.403249 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:56.403147    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:56.476902 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:56.476858    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:56.506314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:56.500699    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:12:56.548312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:56.538428    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:56.658371 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:56.651426    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/health\": context deadline exceeded"
Oct 29 22:12:56.694315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:56.691398    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:12:56.815314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:56.814429    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:57.303339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:57.280033    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:57.441467 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:57.441414    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:57.473994 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:57.441695    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:57.499313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:57.442059    4962 generic.go:356] "Generic (PLEG): container finished" podID="6033cb02-b74f-4cfc-9073-646c4ea795cf" containerID="d2399bc9194752bfb7eaab93c62ab05edc073d29cc2195c9f4163334a7b7a772" exitCode=0
Oct 29 22:12:57.580387 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:57.519715    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:12:57.580387 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:57.523794    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:12:57.580387 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:57.557366    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:57.583549 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:57.583512    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60d52b51-0159-402d-8d66-3c453a0e438f" path="/var/lib/kubelet/pods/60d52b51-0159-402d-8d66-3c453a0e438f/volumes"
Oct 29 22:12:57.610310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:57.584255    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2" event={"ID":"6033cb02-b74f-4cfc-9073-646c4ea795cf","Type":"ContainerDied","Data":"d2399bc9194752bfb7eaab93c62ab05edc073d29cc2195c9f4163334a7b7a772"}
Oct 29 22:12:57.632857 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:57.632815    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 22:12:57.703761 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:57.703728    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:12:57.704037 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:57.704020    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 22:12:57.704185 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:57.704172    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:12:57.704345 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:57.704330    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:12:57.704486 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:57.606646    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:57.828402 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:57.784771    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:12:57.828402 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:57.786514    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:12:57.831527 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:57.831443    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:57.833034 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:57.833008    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:12:58.105388 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:58.093890    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:58.310307 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:58.292641    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:58.631471 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:58.631419    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:58.633950 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:58.633913    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:58.636927 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:58.634658    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:12:58.762437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:58.688027    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:12:58.762437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:58.688360    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:12:58.762437 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:58.688691    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 22:12:58.818311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:58.809503    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:12:58.828370 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:58.819812    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:12:58.861312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:58.832360    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 22:12:58.865025 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:58.864984    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:59.197352 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:59.185598    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg" podUID="70e1bafe-4b93-44d7-a179-44d5a82fd069" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/ready\": context deadline exceeded"
Oct 29 22:12:59.235314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:59.199956    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:59.306416 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:59.306370    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:12:59.463257 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:59.463156    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-fqldg" podUID="70e1bafe-4b93-44d7-a179-44d5a82fd069" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:59.658532 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:59.658490    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:59.659141 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:59.659109    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": context deadline exceeded"
Oct 29 22:12:59.665889 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:59.659724    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:12:59.689694 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:59.689635    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:59.719369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:59.690341    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:12:59.976488 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:59.961040    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:12:59.976488 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:59.962853    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:13:00.004355 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:12:59.985857    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:13:00.108323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:00.099685    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2" podUID="6033cb02-b74f-4cfc-9073-646c4ea795cf" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.174:50052\" within 1s: context deadline exceeded"
Oct 29 22:13:00.117313 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:13:00.112958    4962 logging.go:55] [core] [Channel #3720 SubChannel #3721]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.174:50052", ServerName: "10.132.0.174:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.174:50052: operation was canceled"
Oct 29 22:13:00.202432 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:00.182438    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:13:00.202732 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:00.202697    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr" podUID="10c5ec48-f48a-4703-a82c-9a6995506125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:00.203906 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:00.182771    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:13:00.206380 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:00.206351    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6" podUID="64a76b6c-1542-4fb1-9730-7b770c452b56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:00.609312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:00.600884    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:13:00.609312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:00.600949    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s" podUID="b4cda533-309f-46e1-8cad-7239630ac19d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:00.665555 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:00.664052    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:13:00.665555 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:00.665511    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:13:00.721684 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:00.696772    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:13:00.721684 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:00.699080    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:13:00.721684 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:00.709137    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:00.736654 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:00.736606    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": context deadline exceeded"
Oct 29 22:13:00.814312 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:13:00.813210    4962 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.755s"
Oct 29 22:13:00.851313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:00.844107    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2"
Oct 29 22:13:00.857772 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:00.857726    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded"
Oct 29 22:13:01.107311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:00.910718    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6033cb02-b74f-4cfc-9073-646c4ea795cf-kube-api-access-xvdff\" (UniqueName: \"kubernetes.io/projected/6033cb02-b74f-4cfc-9073-646c4ea795cf-kube-api-access-xvdff\") pod \"6033cb02-b74f-4cfc-9073-646c4ea795cf\" (UID: \"6033cb02-b74f-4cfc-9073-646c4ea795cf\") "
Oct 29 22:13:01.107311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:01.102094    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6033cb02-b74f-4cfc-9073-646c4ea795cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6033cb02-b74f-4cfc-9073-646c4ea795cf-plugins-config-volume\") pod \"6033cb02-b74f-4cfc-9073-646c4ea795cf\" (UID: \"6033cb02-b74f-4cfc-9073-646c4ea795cf\") "
Oct 29 22:13:01.128318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:00.944199    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6033cb02-b74f-4cfc-9073-646c4ea795cf-kube-api-access-xvdff" pod "6033cb02-b74f-4cfc-9073-646c4ea795cf" (UID: "6033cb02-b74f-4cfc-9073-646c4ea795cf"). InnerVolumeSpecName "kube-api-access-xvdff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:01.140315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:00.970917    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:13:01.141597 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:00.970964    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:01.147318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:01.013222    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:01.147609 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:00.976513    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:13:01.147771 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:01.147742    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc" podUID="b72eb074-2855-4dd2-bf0b-e4a42e92d8bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:01.147891 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:01.120479    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6033cb02-b74f-4cfc-9073-646c4ea795cf-plugins-config-volume" pod "6033cb02-b74f-4cfc-9073-646c4ea795cf" (UID: "6033cb02-b74f-4cfc-9073-646c4ea795cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:01.174322 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:00.970825    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:01.240796 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:01.240516    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvdff\" (UniqueName: \"kubernetes.io/projected/6033cb02-b74f-4cfc-9073-646c4ea795cf-kube-api-access-xvdff\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:01.240796 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:01.240571    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6033cb02-b74f-4cfc-9073-646c4ea795cf-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:01.247842 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:01.247813    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2"
Oct 29 22:13:01.399313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:01.397012    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2" event={"ID":"6033cb02-b74f-4cfc-9073-646c4ea795cf","Type":"ContainerDied","Data":"a95fe4900d4a7972d9b06d97ddd1ad431313e1ceba1216b9ad7532fac0903c80"}
Oct 29 22:13:01.399313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:01.399282    4962 scope.go:122] "RemoveContainer" containerID="d2399bc9194752bfb7eaab93c62ab05edc073d29cc2195c9f4163334a7b7a772"
Oct 29 22:13:01.679314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:01.676712    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:01.679314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:01.677116    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:13:01.708313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:01.699299    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:01.708313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:01.699655    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:13:01.744551 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:01.720697    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:01.744551 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:01.742117    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:01.831387 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:01.831343    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:01.974497 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:01.966630    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2"]
Oct 29 22:13:02.031777 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:02.031737    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-75db5b-978fb4c-khzz2"]
Oct 29 22:13:02.143168 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:02.142887    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:02.143711 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:02.143272    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:13:02.155588 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:02.155554    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:02.163401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:02.163368    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:13:02.163615 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:02.163589    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6" podUID="295e3d96-ff42-4be3-b4de-cd85ca4d8578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:02.257472 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:02.257380    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:13:02.257735 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:02.257702    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j" podUID="590ad9cd-3237-4783-8dea-82e67274b580" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:02.306345 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:02.306191    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:13:02.306345 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:02.306255    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z" podUID="132ed775-c9e6-41f1-b224-fff550c6e073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:02.595472 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:02.595374    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:02.680509 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:02.680443    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:13:02.681429 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:02.681387    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:13:02.703480 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:02.703428    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:13:02.704120 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:02.704075    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:02.723114 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:02.723063    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:02.746893 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:02.746755    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:03.150435 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:03.150380    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:03.172313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:03.151466    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:03.199400 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:03.181118    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:13:03.345420 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:03.345378    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6033cb02-b74f-4cfc-9073-646c4ea795cf" path="/var/lib/kubelet/pods/6033cb02-b74f-4cfc-9073-646c4ea795cf/volumes"
Oct 29 22:13:03.735312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:03.734581    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:03.764313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:03.763395    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": context deadline exceeded"
Oct 29 22:13:03.828311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:03.765514    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:03.872726 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:03.768072    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:03.873878 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:03.768821    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:13:03.874803 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:03.768859    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded"
Oct 29 22:13:04.167367 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:04.156123    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:04.184320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:04.182467    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:04.184320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:04.183557    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:04.223211 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:04.223166    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:04.335424 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:04.335374    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:04.735175 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:04.735127    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:13:04.764617 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:04.764573    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:04.908784 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:04.908738    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:04.913550 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:04.909014    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded"
Oct 29 22:13:04.918818 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:04.909043    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:13:04.957425 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:04.909061    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:13:05.211323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:05.194381    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:05.211323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:05.194756    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Oct 29 22:13:05.307332 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:05.304280    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:05.328487 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:05.327911    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:13:05.328487 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:05.327974    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf" podUID="75cf3e4f-0488-4210-ab72-17b7afdeda6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:05.750358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:05.741350    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:13:05.750358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:05.741468    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:13:05.750358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:05.741510    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 22:13:05.750358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:05.741545    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 22:13:05.750358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:05.746524    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:05.776323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:05.767393    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:05.910908 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:05.910868    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:05.927884 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:05.927347    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 22:13:06.105568 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:06.105484    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 22:13:06.195538 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:06.194608    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:13:06.307355 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:06.307312    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:13:06.321189 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:06.321159    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:13:06.352039 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:06.352015    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:13:06.368622 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:06.368562    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:13:09.721628 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:13:09.721586    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:13:10.128884 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.128793    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"]
Oct 29 22:13:10.130326 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.130278    4962 cpu_manager.go:414] "containerMap: removing container" podUID="c1d2bd1d-783b-415a-8807-8f53396ec0ce" containerName="istio-proxy"
Oct 29 22:13:10.130326 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.130324    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d2bd1d-783b-415a-8807-8f53396ec0ce" containerName="istio-proxy"
Oct 29 22:13:10.130492 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.130348    4962 cpu_manager.go:414] "containerMap: removing container" podUID="4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" containerName="istio-proxy"
Oct 29 22:13:10.130492 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.130356    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" containerName="istio-proxy"
Oct 29 22:13:10.130492 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.130380    4962 cpu_manager.go:414] "containerMap: removing container" podUID="4ad0fa83-03aa-4824-ae79-c47cf352ff6a" containerName="istio-proxy"
Oct 29 22:13:10.130492 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.130388    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad0fa83-03aa-4824-ae79-c47cf352ff6a" containerName="istio-proxy"
Oct 29 22:13:10.130492 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.130430    4962 cpu_manager.go:414] "containerMap: removing container" podUID="60d52b51-0159-402d-8d66-3c453a0e438f" containerName="payload-pre-processing"
Oct 29 22:13:10.130492 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.130450    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60d52b51-0159-402d-8d66-3c453a0e438f" containerName="payload-pre-processing"
Oct 29 22:13:10.130492 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.130479    4962 cpu_manager.go:414] "containerMap: removing container" podUID="6033cb02-b74f-4cfc-9073-646c4ea795cf" containerName="payload-pre-processing"
Oct 29 22:13:10.130492 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.130488    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6033cb02-b74f-4cfc-9073-646c4ea795cf" containerName="payload-pre-processing"
Oct 29 22:13:10.130745 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.130675    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d2bd1d-783b-415a-8807-8f53396ec0ce" containerName="istio-proxy"
Oct 29 22:13:10.130745 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.130693    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c84e6b4-e8f7-49b6-ab65-7f677e6fe28a" containerName="istio-proxy"
Oct 29 22:13:10.130745 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.130707    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad0fa83-03aa-4824-ae79-c47cf352ff6a" containerName="istio-proxy"
Oct 29 22:13:10.130745 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.130720    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="60d52b51-0159-402d-8d66-3c453a0e438f" containerName="payload-pre-processing"
Oct 29 22:13:10.130745 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.130736    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="6033cb02-b74f-4cfc-9073-646c4ea795cf" containerName="payload-pre-processing"
Oct 29 22:13:10.175310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.149502    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.223299 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.223074    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-75db5b-openshift-default-dockercfg-mwm8c\""
Oct 29 22:13:10.267323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.265791    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ee7c8e-f519-4574-a3be-f10d70926485-istiod-ca-cert\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.267323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.265957    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-podinfo\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.267323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.265994    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzgfb\" (UniqueName: \"kubernetes.io/projected/a3ee7c8e-f519-4574-a3be-f10d70926485-kube-api-access-rzgfb\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.267323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.266051    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-credential-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.267323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.266514    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-workload-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.267323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.266555    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-data\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.267323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.266850    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-envoy\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.267323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.266933    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-workload-certs\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.267323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.266966    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-token\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.305394 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.305353    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"]
Oct 29 22:13:10.369364 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.369324    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-podinfo\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.369557 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.369375    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzgfb\" (UniqueName: \"kubernetes.io/projected/a3ee7c8e-f519-4574-a3be-f10d70926485-kube-api-access-rzgfb\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.369557 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.369419    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-credential-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.369742 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.369718    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-workload-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.369808 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.369756    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-data\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.369872 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.369837    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-envoy\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.369929 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.369875    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-workload-certs\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.369929 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.369901    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-token\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.370050 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.369931    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ee7c8e-f519-4574-a3be-f10d70926485-istiod-ca-cert\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.370050 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.369942    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-credential-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.370169 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.370142    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-workload-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.370996 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.370228    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-workload-certs\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.370996 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.370448    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-data\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.370996 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.370924    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ee7c8e-f519-4574-a3be-f10d70926485-istiod-ca-cert\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.372841 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.372815    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-envoy\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.372957 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.372816    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-podinfo\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.392079 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.392018    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzgfb\" (UniqueName: \"kubernetes.io/projected/a3ee7c8e-f519-4574-a3be-f10d70926485-kube-api-access-rzgfb\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.392170 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.392125    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-token\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.586239 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.586197    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:10.931198 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.931147    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"]
Oct 29 22:13:10.931685 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:13:10.931569    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ee7c8e_f519_4574_a3be_f10d70926485.slice/crio-7bc6dc7646d805dd7d8db4d5c7b7ec72a29518f553a3440450522702b54f2953 WatchSource:0}: Error finding container 7bc6dc7646d805dd7d8db4d5c7b7ec72a29518f553a3440450522702b54f2953: Status 404 returned error can't find the container with id 7bc6dc7646d805dd7d8db4d5c7b7ec72a29518f553a3440450522702b54f2953
Oct 29 22:13:10.934674 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.934635    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:13:10.934782 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.934700    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:13:10.934782 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:10.934728    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:13:11.834168 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:11.834115    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m" event={"ID":"a3ee7c8e-f519-4574-a3be-f10d70926485","Type":"ContainerStarted","Data":"70ceada97cb370cd59ac50d3ebf826c8f79c5b98646f0ac0eace60ba02c5cdc1"}
Oct 29 22:13:11.834381 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:11.834198    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m" event={"ID":"a3ee7c8e-f519-4574-a3be-f10d70926485","Type":"ContainerStarted","Data":"7bc6dc7646d805dd7d8db4d5c7b7ec72a29518f553a3440450522702b54f2953"}
Oct 29 22:13:11.898160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:11.898092    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m" podStartSLOduration=1.898074083 podStartE2EDuration="1.898074083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:11.893757773 +0000 UTC m=+1612.966714760" watchObservedRunningTime="2026-10-29 22:13:11.898074083 +0000 UTC m=+1612.971031028"
Oct 29 22:13:12.595310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:12.594807    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:12.595310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:12.595001    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:13:12.595310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:12.595068    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m" podUID="a3ee7c8e-f519-4574-a3be-f10d70926485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 29 22:13:13.587358 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:13.587321    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:13:13.587650 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:13.587617    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m" podUID="a3ee7c8e-f519-4574-a3be-f10d70926485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 29 22:13:14.595312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:14.594826    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:13:14.595312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:14.594888    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m" podUID="a3ee7c8e-f519-4574-a3be-f10d70926485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 29 22:13:15.425835 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:15.425796    4962 patch_prober.go:29] interesting pod/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 29 22:13:15.426090 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:15.425858    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl" podUID="093f784b-0baf-4f14-984e-ec8003db30fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 29 22:13:15.450774 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:15.450743    4962 generic.go:356] "Generic (PLEG): container finished" podID="093f784b-0baf-4f14-984e-ec8003db30fb" containerID="c6930380d3da248a77f1ea09cc7b648d3d2f5b54d53a107c727312115a29c31e" exitCode=0
Oct 29 22:13:15.450909 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:15.450798    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl" event={"ID":"093f784b-0baf-4f14-984e-ec8003db30fb","Type":"ContainerDied","Data":"c6930380d3da248a77f1ea09cc7b648d3d2f5b54d53a107c727312115a29c31e"}
Oct 29 22:13:15.451208 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:15.451187    4962 scope.go:122] "RemoveContainer" containerID="c6930380d3da248a77f1ea09cc7b648d3d2f5b54d53a107c727312115a29c31e"
Oct 29 22:13:15.452883 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:15.452854    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:13:15.452947 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:15.452926    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:13:15.452985 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:15.452958    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:13:15.586809 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:15.586704    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:13:15.586809 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:15.586757    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m" podUID="a3ee7c8e-f519-4574-a3be-f10d70926485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 29 22:13:16.459900 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:16.459866    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl" event={"ID":"093f784b-0baf-4f14-984e-ec8003db30fb","Type":"ContainerStarted","Data":"97dd00635c0d257493e656a1cc325cac4e89fb9f39fead1fb6213c33bf7f7299"}
Oct 29 22:13:16.592309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:16.582647    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:13:16.592309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:16.582765    4962 patch_prober.go:29] interesting pod/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 29 22:13:16.592309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:16.582808    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl" podUID="093f784b-0baf-4f14-984e-ec8003db30fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 29 22:13:16.592309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:16.586651    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:13:16.592309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:16.586697    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m" podUID="a3ee7c8e-f519-4574-a3be-f10d70926485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 29 22:13:17.580706 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:17.580668    4962 patch_prober.go:29] interesting pod/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 29 22:13:17.581464 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:17.581429    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl" podUID="093f784b-0baf-4f14-984e-ec8003db30fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 29 22:13:17.590689 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:17.590663    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 29 22:13:17.590865 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:17.590844    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m" podUID="a3ee7c8e-f519-4574-a3be-f10d70926485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 29 22:13:18.596324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:18.596265    4962 patch_prober.go:29] interesting pod/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 29 22:13:18.596731 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:18.596342    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl" podUID="093f784b-0baf-4f14-984e-ec8003db30fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 29 22:13:18.600715 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:18.600693    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:18.601036 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:18.601002    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:18.601817 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:18.601796    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:13:19.595853 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:19.595385    4962 patch_prober.go:29] interesting pod/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 29 22:13:19.595853 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:19.595459    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl" podUID="093f784b-0baf-4f14-984e-ec8003db30fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 29 22:13:20.582640 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:20.582602    4962 patch_prober.go:29] interesting pod/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 29 22:13:20.583254 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:20.583222    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl" podUID="093f784b-0baf-4f14-984e-ec8003db30fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 29 22:13:21.589329 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:21.585495    4962 patch_prober.go:29] interesting pod/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 29 22:13:21.589329 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:21.585574    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl" podUID="093f784b-0baf-4f14-984e-ec8003db30fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 29 22:13:21.804404 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:21.804337    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8"]
Oct 29 22:13:21.804914 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:21.804859    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8" podUID="df517504-590f-4f2b-8ab1-5e92e1f1936c" containerName="maas-api" containerID="cri-o://e3e683c9d02b3999bd45242baddc3f1ffe93d5216cd33fd4582a71edd8c04302" gracePeriod=30
Oct 29 22:13:21.952504 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:21.952437    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r"]
Oct 29 22:13:21.952748 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:21.952714    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r" podUID="4c2c8ef0-de16-4e8c-b524-21e8151dc910" containerName="payload-pre-processing" containerID="cri-o://085bd13306ed47eac1f4efa4749cf1048de6f14c8db32d7d61872d7db75fbecf" gracePeriod=30
Oct 29 22:13:22.411313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.403258    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8"
Oct 29 22:13:22.584477 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.583975    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df517504-590f-4f2b-8ab1-5e92e1f1936c-kube-api-access-ktk5g\" (UniqueName: \"kubernetes.io/projected/df517504-590f-4f2b-8ab1-5e92e1f1936c-kube-api-access-ktk5g\") pod \"df517504-590f-4f2b-8ab1-5e92e1f1936c\" (UID: \"df517504-590f-4f2b-8ab1-5e92e1f1936c\") "
Oct 29 22:13:22.584477 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.584049    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df517504-590f-4f2b-8ab1-5e92e1f1936c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df517504-590f-4f2b-8ab1-5e92e1f1936c-metrics-certs\") pod \"df517504-590f-4f2b-8ab1-5e92e1f1936c\" (UID: \"df517504-590f-4f2b-8ab1-5e92e1f1936c\") "
Oct 29 22:13:22.584477 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.584130    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df517504-590f-4f2b-8ab1-5e92e1f1936c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df517504-590f-4f2b-8ab1-5e92e1f1936c-maas-api-tls\") pod \"df517504-590f-4f2b-8ab1-5e92e1f1936c\" (UID: \"df517504-590f-4f2b-8ab1-5e92e1f1936c\") "
Oct 29 22:13:22.584477 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.584259    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df517504-590f-4f2b-8ab1-5e92e1f1936c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df517504-590f-4f2b-8ab1-5e92e1f1936c-openshift-service-ca\") pod \"df517504-590f-4f2b-8ab1-5e92e1f1936c\" (UID: \"df517504-590f-4f2b-8ab1-5e92e1f1936c\") "
Oct 29 22:13:22.602249 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.602210    4962 patch_prober.go:29] interesting pod/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Oct 29 22:13:22.611324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.602869    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl" podUID="093f784b-0baf-4f14-984e-ec8003db30fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Oct 29 22:13:22.611324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.608517    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df517504-590f-4f2b-8ab1-5e92e1f1936c-openshift-service-ca" pod "df517504-590f-4f2b-8ab1-5e92e1f1936c" (UID: "df517504-590f-4f2b-8ab1-5e92e1f1936c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:22.659126 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.659060    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df517504-590f-4f2b-8ab1-5e92e1f1936c-kube-api-access-ktk5g" pod "df517504-590f-4f2b-8ab1-5e92e1f1936c" (UID: "df517504-590f-4f2b-8ab1-5e92e1f1936c"). InnerVolumeSpecName "kube-api-access-ktk5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:22.659126 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.659075    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df517504-590f-4f2b-8ab1-5e92e1f1936c-maas-api-tls" pod "df517504-590f-4f2b-8ab1-5e92e1f1936c" (UID: "df517504-590f-4f2b-8ab1-5e92e1f1936c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:13:22.692496 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.681479    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df517504-590f-4f2b-8ab1-5e92e1f1936c-metrics-certs" pod "df517504-590f-4f2b-8ab1-5e92e1f1936c" (UID: "df517504-590f-4f2b-8ab1-5e92e1f1936c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:13:22.706615 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.699374    4962 generic.go:356] "Generic (PLEG): container finished" podID="df517504-590f-4f2b-8ab1-5e92e1f1936c" containerID="e3e683c9d02b3999bd45242baddc3f1ffe93d5216cd33fd4582a71edd8c04302" exitCode=0
Oct 29 22:13:22.706615 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.699511    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8" event={"ID":"df517504-590f-4f2b-8ab1-5e92e1f1936c","Type":"ContainerDied","Data":"e3e683c9d02b3999bd45242baddc3f1ffe93d5216cd33fd4582a71edd8c04302"}
Oct 29 22:13:22.706615 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.699586    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8" event={"ID":"df517504-590f-4f2b-8ab1-5e92e1f1936c","Type":"ContainerDied","Data":"af61d17f1c3d952876a4223053d9738f653f3b76b76eec92da0e78644e20373d"}
Oct 29 22:13:22.706615 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.699617    4962 scope.go:122] "RemoveContainer" containerID="e3e683c9d02b3999bd45242baddc3f1ffe93d5216cd33fd4582a71edd8c04302"
Oct 29 22:13:22.706615 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.699769    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8"
Oct 29 22:13:22.706615 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.704595    4962 generic.go:356] "Generic (PLEG): container finished" podID="4c2c8ef0-de16-4e8c-b524-21e8151dc910" containerID="085bd13306ed47eac1f4efa4749cf1048de6f14c8db32d7d61872d7db75fbecf" exitCode=0
Oct 29 22:13:22.706615 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.704760    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r" event={"ID":"4c2c8ef0-de16-4e8c-b524-21e8151dc910","Type":"ContainerDied","Data":"085bd13306ed47eac1f4efa4749cf1048de6f14c8db32d7d61872d7db75fbecf"}
Oct 29 22:13:22.714566 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.713611    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktk5g\" (UniqueName: \"kubernetes.io/projected/df517504-590f-4f2b-8ab1-5e92e1f1936c-kube-api-access-ktk5g\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:22.714566 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.713633    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df517504-590f-4f2b-8ab1-5e92e1f1936c-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:22.714566 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.713643    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df517504-590f-4f2b-8ab1-5e92e1f1936c-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:22.714566 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.713651    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df517504-590f-4f2b-8ab1-5e92e1f1936c-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:22.728934 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.728901    4962 scope.go:122] "RemoveContainer" containerID="e3e683c9d02b3999bd45242baddc3f1ffe93d5216cd33fd4582a71edd8c04302"
Oct 29 22:13:22.729447 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:13:22.729417    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3e683c9d02b3999bd45242baddc3f1ffe93d5216cd33fd4582a71edd8c04302\": container with ID starting with e3e683c9d02b3999bd45242baddc3f1ffe93d5216cd33fd4582a71edd8c04302 not found: ID does not exist" containerID="e3e683c9d02b3999bd45242baddc3f1ffe93d5216cd33fd4582a71edd8c04302"
Oct 29 22:13:22.729537 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.729462    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3e683c9d02b3999bd45242baddc3f1ffe93d5216cd33fd4582a71edd8c04302"} err="failed to get container status \"e3e683c9d02b3999bd45242baddc3f1ffe93d5216cd33fd4582a71edd8c04302\": rpc error: code = NotFound desc = could not find container \"e3e683c9d02b3999bd45242baddc3f1ffe93d5216cd33fd4582a71edd8c04302\": container with ID starting with e3e683c9d02b3999bd45242baddc3f1ffe93d5216cd33fd4582a71edd8c04302 not found: ID does not exist"
Oct 29 22:13:22.752469 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.752447    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r"
Oct 29 22:13:22.763257 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.763233    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8"]
Oct 29 22:13:22.770269 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.770226    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f82846-ccb54cd77-7sjb8"]
Oct 29 22:13:22.863000 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.855760    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c2c8ef0-de16-4e8c-b524-21e8151dc910-kube-api-access-cgfnx\" (UniqueName: \"kubernetes.io/projected/4c2c8ef0-de16-4e8c-b524-21e8151dc910-kube-api-access-cgfnx\") pod \"4c2c8ef0-de16-4e8c-b524-21e8151dc910\" (UID: \"4c2c8ef0-de16-4e8c-b524-21e8151dc910\") "
Oct 29 22:13:22.863000 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.856016    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c2c8ef0-de16-4e8c-b524-21e8151dc910-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2c8ef0-de16-4e8c-b524-21e8151dc910-plugins-config-volume\") pod \"4c2c8ef0-de16-4e8c-b524-21e8151dc910\" (UID: \"4c2c8ef0-de16-4e8c-b524-21e8151dc910\") "
Oct 29 22:13:22.898392 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.887649    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c2c8ef0-de16-4e8c-b524-21e8151dc910-plugins-config-volume" pod "4c2c8ef0-de16-4e8c-b524-21e8151dc910" (UID: "4c2c8ef0-de16-4e8c-b524-21e8151dc910"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:22.906312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.906254    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2c8ef0-de16-4e8c-b524-21e8151dc910-kube-api-access-cgfnx" pod "4c2c8ef0-de16-4e8c-b524-21e8151dc910" (UID: "4c2c8ef0-de16-4e8c-b524-21e8151dc910"). InnerVolumeSpecName "kube-api-access-cgfnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:22.971688 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.971646    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgfnx\" (UniqueName: \"kubernetes.io/projected/4c2c8ef0-de16-4e8c-b524-21e8151dc910-kube-api-access-cgfnx\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:22.973399 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:22.973368    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2c8ef0-de16-4e8c-b524-21e8151dc910-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:23.060927 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:23.060892    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df517504-590f-4f2b-8ab1-5e92e1f1936c" path="/var/lib/kubelet/pods/df517504-590f-4f2b-8ab1-5e92e1f1936c/volumes"
Oct 29 22:13:23.584612 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:23.584584    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:13:23.585048 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:23.585033    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:13:23.585678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:23.585656    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:13:23.718048 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:23.718022    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r"
Oct 29 22:13:23.718687 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:23.718040    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r" event={"ID":"4c2c8ef0-de16-4e8c-b524-21e8151dc910","Type":"ContainerDied","Data":"b942c084ee07b91ae53ca267d3039fb2ef839dae7594c5678c5e6c4dba9c9be1"}
Oct 29 22:13:23.718818 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:23.718715    4962 scope.go:122] "RemoveContainer" containerID="085bd13306ed47eac1f4efa4749cf1048de6f14c8db32d7d61872d7db75fbecf"
Oct 29 22:13:23.772756 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:23.772728    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r"]
Oct 29 22:13:23.776668 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:23.776643    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f82846-f95f6956c-pxr7r"]
Oct 29 22:13:25.060585 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:25.060545    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c2c8ef0-de16-4e8c-b524-21e8151dc910" path="/var/lib/kubelet/pods/4c2c8ef0-de16-4e8c-b524-21e8151dc910/volumes"
Oct 29 22:13:30.433311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:30.433070    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"]
Oct 29 22:13:30.433828 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:30.433446    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm" podUID="faea6cb2-afaf-49a8-81ab-bb9c3f699e35" containerName="istio-proxy" containerID="cri-o://81135eb7acd2c9858e998d0b27e6fcbdaaa553970f782deec510669bf5b1f318" gracePeriod=30
Oct 29 22:13:31.470589 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:31.470556    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj"]
Oct 29 22:13:31.502158 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:31.502120    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx"]
Oct 29 22:13:35.609786 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:35.608365    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"]
Oct 29 22:13:35.609786 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:35.608655    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp" podUID="5e577b68-1188-430f-9930-eb43bc82dd07" containerName="maas-api" containerID="cri-o://0cecb9a41275548a1b3e011afdeeab9091e9e79bd6f49ba10c3aae8ea96f0e1f" gracePeriod=30
Oct 29 22:13:35.803646 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:35.803613    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:13:35.907312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:35.888326    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-kube-api-access-q4q9f\" (UniqueName: \"kubernetes.io/projected/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-kube-api-access-q4q9f\") pod \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") "
Oct 29 22:13:35.907312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:35.888423    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-envoy\") pod \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") "
Oct 29 22:13:35.907312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:35.888485    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-token\" (UniqueName: \"kubernetes.io/projected/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-token\") pod \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") "
Oct 29 22:13:35.907312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:35.888532    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-workload-socket\") pod \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") "
Oct 29 22:13:35.907312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:35.888683    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-podinfo\") pod \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") "
Oct 29 22:13:35.907312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:35.888838    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-credential-socket\") pod \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") "
Oct 29 22:13:35.907312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:35.888902    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istiod-ca-cert\") pod \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") "
Oct 29 22:13:35.907312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:35.889008    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-workload-certs\") pod \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") "
Oct 29 22:13:35.907312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:35.889131    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-data\") pod \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\" (UID: \"faea6cb2-afaf-49a8-81ab-bb9c3f699e35\") "
Oct 29 22:13:35.942843 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:35.942408    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-workload-socket" pod "faea6cb2-afaf-49a8-81ab-bb9c3f699e35" (UID: "faea6cb2-afaf-49a8-81ab-bb9c3f699e35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:36.008365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.005819    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-data" pod "faea6cb2-afaf-49a8-81ab-bb9c3f699e35" (UID: "faea6cb2-afaf-49a8-81ab-bb9c3f699e35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:36.080315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.035478    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-kube-api-access-q4q9f" pod "faea6cb2-afaf-49a8-81ab-bb9c3f699e35" (UID: "faea6cb2-afaf-49a8-81ab-bb9c3f699e35"). InnerVolumeSpecName "kube-api-access-q4q9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:36.080315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.046503    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-credential-socket" pod "faea6cb2-afaf-49a8-81ab-bb9c3f699e35" (UID: "faea6cb2-afaf-49a8-81ab-bb9c3f699e35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:36.080315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.046602    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-workload-certs" pod "faea6cb2-afaf-49a8-81ab-bb9c3f699e35" (UID: "faea6cb2-afaf-49a8-81ab-bb9c3f699e35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:36.080315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.046751    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istiod-ca-cert" pod "faea6cb2-afaf-49a8-81ab-bb9c3f699e35" (UID: "faea6cb2-afaf-49a8-81ab-bb9c3f699e35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:36.080315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.049389    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.080315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.049434    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.189301 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.130421    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"]
Oct 29 22:13:36.270360 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.200354    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.270360 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.200397    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.270360 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.200412    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.270360 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.200424    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4q9f\" (UniqueName: \"kubernetes.io/projected/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-kube-api-access-q4q9f\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.296626 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.292478    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-envoy" pod "faea6cb2-afaf-49a8-81ab-bb9c3f699e35" (UID: "faea6cb2-afaf-49a8-81ab-bb9c3f699e35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:36.350697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.339239    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.361373 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.355635    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-podinfo" pod "faea6cb2-afaf-49a8-81ab-bb9c3f699e35" (UID: "faea6cb2-afaf-49a8-81ab-bb9c3f699e35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:13:36.459311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.455894    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.470270 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.470235    4962 generic.go:356] "Generic (PLEG): container finished" podID="5e577b68-1188-430f-9930-eb43bc82dd07" containerID="0cecb9a41275548a1b3e011afdeeab9091e9e79bd6f49ba10c3aae8ea96f0e1f" exitCode=0
Oct 29 22:13:36.470694 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.470667    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp" event={"ID":"5e577b68-1188-430f-9930-eb43bc82dd07","Type":"ContainerDied","Data":"0cecb9a41275548a1b3e011afdeeab9091e9e79bd6f49ba10c3aae8ea96f0e1f"}
Oct 29 22:13:36.470897 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.470877    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj" podUID="0897e113-b10f-460b-9b0f-17ce9daea2c7" containerName="payload-processing" containerID="cri-o://59aa8976d5f4d879aa9f6c90c864bd8c434090e97702b617a871d7814063a3af" gracePeriod=25
Oct 29 22:13:36.522396 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.509183    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx" podUID="68dfa23b-de59-4864-a08e-7ab7185792a8" containerName="payload-pre-processing" containerID="cri-o://4419e1615d1af83169c7ff79d7d55a8f443b7720f986d8de658ffa33f69e3c7a" gracePeriod=25
Oct 29 22:13:36.551361 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.520048    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-token" pod "faea6cb2-afaf-49a8-81ab-bb9c3f699e35" (UID: "faea6cb2-afaf-49a8-81ab-bb9c3f699e35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:36.551361 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.520428    4962 generic.go:356] "Generic (PLEG): container finished" podID="faea6cb2-afaf-49a8-81ab-bb9c3f699e35" containerID="81135eb7acd2c9858e998d0b27e6fcbdaaa553970f782deec510669bf5b1f318" exitCode=0
Oct 29 22:13:36.551361 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.520468    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm" event={"ID":"faea6cb2-afaf-49a8-81ab-bb9c3f699e35","Type":"ContainerDied","Data":"81135eb7acd2c9858e998d0b27e6fcbdaaa553970f782deec510669bf5b1f318"}
Oct 29 22:13:36.551361 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.524360    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm" event={"ID":"faea6cb2-afaf-49a8-81ab-bb9c3f699e35","Type":"ContainerDied","Data":"5692a96c5ff999f2e5aba13b787e198a6e342e104a1f220d35527c7804097022"}
Oct 29 22:13:36.551361 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.524398    4962 scope.go:122] "RemoveContainer" containerID="81135eb7acd2c9858e998d0b27e6fcbdaaa553970f782deec510669bf5b1f318"
Oct 29 22:13:36.551361 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.520556    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"
Oct 29 22:13:36.560350 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.560280    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faea6cb2-afaf-49a8-81ab-bb9c3f699e35-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.572421 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.572390    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"]
Oct 29 22:13:36.578208 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.578176    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"]
Oct 29 22:13:36.713165 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.713144    4962 scope.go:122] "RemoveContainer" containerID="44c462ca217edfa89d757be4543e0b8ba0dc40798afa07384667979bc9d4b870"
Oct 29 22:13:36.718489 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.718467    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"
Oct 29 22:13:36.724692 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.724674    4962 scope.go:122] "RemoveContainer" containerID="81135eb7acd2c9858e998d0b27e6fcbdaaa553970f782deec510669bf5b1f318"
Oct 29 22:13:36.725019 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:13:36.724991    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81135eb7acd2c9858e998d0b27e6fcbdaaa553970f782deec510669bf5b1f318\": container with ID starting with 81135eb7acd2c9858e998d0b27e6fcbdaaa553970f782deec510669bf5b1f318 not found: ID does not exist" containerID="81135eb7acd2c9858e998d0b27e6fcbdaaa553970f782deec510669bf5b1f318"
Oct 29 22:13:36.725079 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.725031    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81135eb7acd2c9858e998d0b27e6fcbdaaa553970f782deec510669bf5b1f318"} err="failed to get container status \"81135eb7acd2c9858e998d0b27e6fcbdaaa553970f782deec510669bf5b1f318\": rpc error: code = NotFound desc = could not find container \"81135eb7acd2c9858e998d0b27e6fcbdaaa553970f782deec510669bf5b1f318\": container with ID starting with 81135eb7acd2c9858e998d0b27e6fcbdaaa553970f782deec510669bf5b1f318 not found: ID does not exist"
Oct 29 22:13:36.725079 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.725053    4962 scope.go:122] "RemoveContainer" containerID="44c462ca217edfa89d757be4543e0b8ba0dc40798afa07384667979bc9d4b870"
Oct 29 22:13:36.725439 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:13:36.725414    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44c462ca217edfa89d757be4543e0b8ba0dc40798afa07384667979bc9d4b870\": container with ID starting with 44c462ca217edfa89d757be4543e0b8ba0dc40798afa07384667979bc9d4b870 not found: ID does not exist" containerID="44c462ca217edfa89d757be4543e0b8ba0dc40798afa07384667979bc9d4b870"
Oct 29 22:13:36.725492 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.725447    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44c462ca217edfa89d757be4543e0b8ba0dc40798afa07384667979bc9d4b870"} err="failed to get container status \"44c462ca217edfa89d757be4543e0b8ba0dc40798afa07384667979bc9d4b870\": rpc error: code = NotFound desc = could not find container \"44c462ca217edfa89d757be4543e0b8ba0dc40798afa07384667979bc9d4b870\": container with ID starting with 44c462ca217edfa89d757be4543e0b8ba0dc40798afa07384667979bc9d4b870 not found: ID does not exist"
Oct 29 22:13:36.753998 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.753969    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"]
Oct 29 22:13:36.763846 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.762174    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f82846-openshift-default-5cb5cd7b65-h2wcm"]
Oct 29 22:13:36.763846 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.763242    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e577b68-1188-430f-9930-eb43bc82dd07-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e577b68-1188-430f-9930-eb43bc82dd07-openshift-service-ca\") pod \"5e577b68-1188-430f-9930-eb43bc82dd07\" (UID: \"5e577b68-1188-430f-9930-eb43bc82dd07\") "
Oct 29 22:13:36.764076 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.763969    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e577b68-1188-430f-9930-eb43bc82dd07-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e577b68-1188-430f-9930-eb43bc82dd07-metrics-certs\") pod \"5e577b68-1188-430f-9930-eb43bc82dd07\" (UID: \"5e577b68-1188-430f-9930-eb43bc82dd07\") "
Oct 29 22:13:36.764076 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.764051    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e577b68-1188-430f-9930-eb43bc82dd07-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e577b68-1188-430f-9930-eb43bc82dd07-maas-api-tls\") pod \"5e577b68-1188-430f-9930-eb43bc82dd07\" (UID: \"5e577b68-1188-430f-9930-eb43bc82dd07\") "
Oct 29 22:13:36.764195 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.764178    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e577b68-1188-430f-9930-eb43bc82dd07-kube-api-access-btcr6\" (UniqueName: \"kubernetes.io/projected/5e577b68-1188-430f-9930-eb43bc82dd07-kube-api-access-btcr6\") pod \"5e577b68-1188-430f-9930-eb43bc82dd07\" (UID: \"5e577b68-1188-430f-9930-eb43bc82dd07\") "
Oct 29 22:13:36.765393 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.764681    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e577b68-1188-430f-9930-eb43bc82dd07-openshift-service-ca" pod "5e577b68-1188-430f-9930-eb43bc82dd07" (UID: "5e577b68-1188-430f-9930-eb43bc82dd07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:36.765393 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.765086    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e577b68-1188-430f-9930-eb43bc82dd07-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.767386 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.767344    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e577b68-1188-430f-9930-eb43bc82dd07-maas-api-tls" pod "5e577b68-1188-430f-9930-eb43bc82dd07" (UID: "5e577b68-1188-430f-9930-eb43bc82dd07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:13:36.768258 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.768230    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e577b68-1188-430f-9930-eb43bc82dd07-metrics-certs" pod "5e577b68-1188-430f-9930-eb43bc82dd07" (UID: "5e577b68-1188-430f-9930-eb43bc82dd07"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:13:36.769151 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.769125    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e577b68-1188-430f-9930-eb43bc82dd07-kube-api-access-btcr6" pod "5e577b68-1188-430f-9930-eb43bc82dd07" (UID: "5e577b68-1188-430f-9930-eb43bc82dd07"). InnerVolumeSpecName "kube-api-access-btcr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:36.844401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.844371    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx"
Oct 29 22:13:36.866195 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.866169    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68dfa23b-de59-4864-a08e-7ab7185792a8-kube-api-access-4szv9\" (UniqueName: \"kubernetes.io/projected/68dfa23b-de59-4864-a08e-7ab7185792a8-kube-api-access-4szv9\") pod \"68dfa23b-de59-4864-a08e-7ab7185792a8\" (UID: \"68dfa23b-de59-4864-a08e-7ab7185792a8\") "
Oct 29 22:13:36.866342 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.866305    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68dfa23b-de59-4864-a08e-7ab7185792a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68dfa23b-de59-4864-a08e-7ab7185792a8-plugins-config-volume\") pod \"68dfa23b-de59-4864-a08e-7ab7185792a8\" (UID: \"68dfa23b-de59-4864-a08e-7ab7185792a8\") "
Oct 29 22:13:36.866728 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.866695    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68dfa23b-de59-4864-a08e-7ab7185792a8-plugins-config-volume" pod "68dfa23b-de59-4864-a08e-7ab7185792a8" (UID: "68dfa23b-de59-4864-a08e-7ab7185792a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:36.867015 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.866996    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btcr6\" (UniqueName: \"kubernetes.io/projected/5e577b68-1188-430f-9930-eb43bc82dd07-kube-api-access-btcr6\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.867062 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.867022    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68dfa23b-de59-4864-a08e-7ab7185792a8-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.867062 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.867038    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e577b68-1188-430f-9930-eb43bc82dd07-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.867062 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.867051    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e577b68-1188-430f-9930-eb43bc82dd07-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:36.868608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:36.868584    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68dfa23b-de59-4864-a08e-7ab7185792a8-kube-api-access-4szv9" pod "68dfa23b-de59-4864-a08e-7ab7185792a8" (UID: "68dfa23b-de59-4864-a08e-7ab7185792a8"). InnerVolumeSpecName "kube-api-access-4szv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:37.016925 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.016532    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj"
Oct 29 22:13:37.017549 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.017478    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4szv9\" (UniqueName: \"kubernetes.io/projected/68dfa23b-de59-4864-a08e-7ab7185792a8-kube-api-access-4szv9\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:37.063395 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.063270    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faea6cb2-afaf-49a8-81ab-bb9c3f699e35" path="/var/lib/kubelet/pods/faea6cb2-afaf-49a8-81ab-bb9c3f699e35/volumes"
Oct 29 22:13:37.118337 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.118308    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0897e113-b10f-460b-9b0f-17ce9daea2c7-kube-api-access-7rl5c\" (UniqueName: \"kubernetes.io/projected/0897e113-b10f-460b-9b0f-17ce9daea2c7-kube-api-access-7rl5c\") pod \"0897e113-b10f-460b-9b0f-17ce9daea2c7\" (UID: \"0897e113-b10f-460b-9b0f-17ce9daea2c7\") "
Oct 29 22:13:37.118503 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.118355    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0897e113-b10f-460b-9b0f-17ce9daea2c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0897e113-b10f-460b-9b0f-17ce9daea2c7-plugins-config-volume\") pod \"0897e113-b10f-460b-9b0f-17ce9daea2c7\" (UID: \"0897e113-b10f-460b-9b0f-17ce9daea2c7\") "
Oct 29 22:13:37.118730 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.118707    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0897e113-b10f-460b-9b0f-17ce9daea2c7-plugins-config-volume" pod "0897e113-b10f-460b-9b0f-17ce9daea2c7" (UID: "0897e113-b10f-460b-9b0f-17ce9daea2c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:37.119603 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.119576    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0897e113-b10f-460b-9b0f-17ce9daea2c7-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:37.120782 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.120757    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0897e113-b10f-460b-9b0f-17ce9daea2c7-kube-api-access-7rl5c" pod "0897e113-b10f-460b-9b0f-17ce9daea2c7" (UID: "0897e113-b10f-460b-9b0f-17ce9daea2c7"). InnerVolumeSpecName "kube-api-access-7rl5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:37.220938 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.220912    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rl5c\" (UniqueName: \"kubernetes.io/projected/0897e113-b10f-460b-9b0f-17ce9daea2c7-kube-api-access-7rl5c\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:37.540717 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.540688    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"
Oct 29 22:13:37.540880 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.540686    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp" event={"ID":"5e577b68-1188-430f-9930-eb43bc82dd07","Type":"ContainerDied","Data":"75de97a8d95537ba59eec77968856a9f5abb292e55b648641a387a96d22831dd"}
Oct 29 22:13:37.540880 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.540862    4962 scope.go:122] "RemoveContainer" containerID="0cecb9a41275548a1b3e011afdeeab9091e9e79bd6f49ba10c3aae8ea96f0e1f"
Oct 29 22:13:37.542755 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.542729    4962 generic.go:356] "Generic (PLEG): container finished" podID="0897e113-b10f-460b-9b0f-17ce9daea2c7" containerID="59aa8976d5f4d879aa9f6c90c864bd8c434090e97702b617a871d7814063a3af" exitCode=0
Oct 29 22:13:37.542924 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.542788    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj"
Oct 29 22:13:37.542924 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.542876    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj" event={"ID":"0897e113-b10f-460b-9b0f-17ce9daea2c7","Type":"ContainerDied","Data":"59aa8976d5f4d879aa9f6c90c864bd8c434090e97702b617a871d7814063a3af"}
Oct 29 22:13:37.542924 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.542913    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj" event={"ID":"0897e113-b10f-460b-9b0f-17ce9daea2c7","Type":"ContainerDied","Data":"1a093c0f6cbde5f01ca4f6c06fc6ee29912409dad53d507a4e71b9e15d656257"}
Oct 29 22:13:37.546086 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.546060    4962 generic.go:356] "Generic (PLEG): container finished" podID="68dfa23b-de59-4864-a08e-7ab7185792a8" containerID="4419e1615d1af83169c7ff79d7d55a8f443b7720f986d8de658ffa33f69e3c7a" exitCode=0
Oct 29 22:13:37.546202 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.546126    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx"
Oct 29 22:13:37.546202 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.546177    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx" event={"ID":"68dfa23b-de59-4864-a08e-7ab7185792a8","Type":"ContainerDied","Data":"4419e1615d1af83169c7ff79d7d55a8f443b7720f986d8de658ffa33f69e3c7a"}
Oct 29 22:13:37.546340 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.546219    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx" event={"ID":"68dfa23b-de59-4864-a08e-7ab7185792a8","Type":"ContainerDied","Data":"928a91dc7828cf0986927727d63784c761b4e64f9e0a80947abbd8506cee18b2"}
Oct 29 22:13:37.553100 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.553084    4962 scope.go:122] "RemoveContainer" containerID="59aa8976d5f4d879aa9f6c90c864bd8c434090e97702b617a871d7814063a3af"
Oct 29 22:13:37.562568 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.562390    4962 scope.go:122] "RemoveContainer" containerID="59aa8976d5f4d879aa9f6c90c864bd8c434090e97702b617a871d7814063a3af"
Oct 29 22:13:37.562646 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:13:37.562628    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59aa8976d5f4d879aa9f6c90c864bd8c434090e97702b617a871d7814063a3af\": container with ID starting with 59aa8976d5f4d879aa9f6c90c864bd8c434090e97702b617a871d7814063a3af not found: ID does not exist" containerID="59aa8976d5f4d879aa9f6c90c864bd8c434090e97702b617a871d7814063a3af"
Oct 29 22:13:37.562684 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.562653    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59aa8976d5f4d879aa9f6c90c864bd8c434090e97702b617a871d7814063a3af"} err="failed to get container status \"59aa8976d5f4d879aa9f6c90c864bd8c434090e97702b617a871d7814063a3af\": rpc error: code = NotFound desc = could not find container \"59aa8976d5f4d879aa9f6c90c864bd8c434090e97702b617a871d7814063a3af\": container with ID starting with 59aa8976d5f4d879aa9f6c90c864bd8c434090e97702b617a871d7814063a3af not found: ID does not exist"
Oct 29 22:13:37.562684 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.562668    4962 scope.go:122] "RemoveContainer" containerID="4419e1615d1af83169c7ff79d7d55a8f443b7720f986d8de658ffa33f69e3c7a"
Oct 29 22:13:37.572032 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.572011    4962 scope.go:122] "RemoveContainer" containerID="4419e1615d1af83169c7ff79d7d55a8f443b7720f986d8de658ffa33f69e3c7a"
Oct 29 22:13:37.572321 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:13:37.572272    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4419e1615d1af83169c7ff79d7d55a8f443b7720f986d8de658ffa33f69e3c7a\": container with ID starting with 4419e1615d1af83169c7ff79d7d55a8f443b7720f986d8de658ffa33f69e3c7a not found: ID does not exist" containerID="4419e1615d1af83169c7ff79d7d55a8f443b7720f986d8de658ffa33f69e3c7a"
Oct 29 22:13:37.572395 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.572309    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4419e1615d1af83169c7ff79d7d55a8f443b7720f986d8de658ffa33f69e3c7a"} err="failed to get container status \"4419e1615d1af83169c7ff79d7d55a8f443b7720f986d8de658ffa33f69e3c7a\": rpc error: code = NotFound desc = could not find container \"4419e1615d1af83169c7ff79d7d55a8f443b7720f986d8de658ffa33f69e3c7a\": container with ID starting with 4419e1615d1af83169c7ff79d7d55a8f443b7720f986d8de658ffa33f69e3c7a not found: ID does not exist"
Oct 29 22:13:37.591837 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.591817    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"]
Oct 29 22:13:37.603533 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.603511    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-48c52b-864449db78-t4tqp"]
Oct 29 22:13:37.635040 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.635014    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx"]
Oct 29 22:13:37.642486 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.642465    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4e8583-898d6767-2nbqx"]
Oct 29 22:13:37.679751 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.679733    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj"]
Oct 29 22:13:37.683382 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:37.683363    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4e8583-d95bddb4d-zlfpj"]
Oct 29 22:13:39.059742 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:39.059710    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0897e113-b10f-460b-9b0f-17ce9daea2c7" path="/var/lib/kubelet/pods/0897e113-b10f-460b-9b0f-17ce9daea2c7/volumes"
Oct 29 22:13:39.060182 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:39.060168    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e577b68-1188-430f-9930-eb43bc82dd07" path="/var/lib/kubelet/pods/5e577b68-1188-430f-9930-eb43bc82dd07/volumes"
Oct 29 22:13:39.060628 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:39.060612    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68dfa23b-de59-4864-a08e-7ab7185792a8" path="/var/lib/kubelet/pods/68dfa23b-de59-4864-a08e-7ab7185792a8/volumes"
Oct 29 22:13:39.728200 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:13:39.728155    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:13:44.308309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:44.305236    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44"]
Oct 29 22:13:44.308309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:44.305514    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44" podUID="674cc938-504c-4547-abb7-cc2ee413d6cf" containerName="maas-api" containerID="cri-o://76e8b0fde55763dad76d081e0cae0454330942ca4717ffa626a854c256925a63" gracePeriod=30
Oct 29 22:13:44.468114 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:44.467426    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"]
Oct 29 22:13:44.468114 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:44.467658    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p" podUID="13d62170-e1c5-4478-886e-1b7b7f20d9e0" containerName="istio-proxy" containerID="cri-o://8c351b4f6ebf661af34efaf64ed4ae5a99c86b78b241f60863c7c73b89a2a906" gracePeriod=30
Oct 29 22:13:44.480679 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:44.480645    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"]
Oct 29 22:13:44.481189 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:44.480923    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv" podUID="654ebdbf-c9c1-4571-91b1-465aa5840051" containerName="istio-proxy" containerID="cri-o://f686bd197707f6a55bf13ee343a2b4fb3d69320293fe1ba76fb230f5b7f6159c" gracePeriod=30
Oct 29 22:13:44.486137 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:44.486112    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"]
Oct 29 22:13:44.486413 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:44.486372    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg" podUID="be60e8e6-3cc5-45f2-b8f0-0d55d04e490a" containerName="istio-proxy" containerID="cri-o://601e860facd433df5ae34264a45424e57f7390f9743d15dcbdc7d7bdd7fd9014" gracePeriod=30
Oct 29 22:13:44.646741 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:44.646713    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44"
Oct 29 22:13:44.647603 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:44.647574    4962 generic.go:356] "Generic (PLEG): container finished" podID="674cc938-504c-4547-abb7-cc2ee413d6cf" containerID="76e8b0fde55763dad76d081e0cae0454330942ca4717ffa626a854c256925a63" exitCode=0
Oct 29 22:13:44.647724 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:44.647648    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44" event={"ID":"674cc938-504c-4547-abb7-cc2ee413d6cf","Type":"ContainerDied","Data":"76e8b0fde55763dad76d081e0cae0454330942ca4717ffa626a854c256925a63"}
Oct 29 22:13:44.647724 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:44.647715    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44" event={"ID":"674cc938-504c-4547-abb7-cc2ee413d6cf","Type":"ContainerDied","Data":"f0ae3ab952429c68fc69b2841cf979f2f61d1493d1c65e8a4410503adf29d4f6"}
Oct 29 22:13:44.647818 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:44.647745    4962 scope.go:122] "RemoveContainer" containerID="76e8b0fde55763dad76d081e0cae0454330942ca4717ffa626a854c256925a63"
Oct 29 22:13:44.753279 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:44.747594    4962 scope.go:122] "RemoveContainer" containerID="76e8b0fde55763dad76d081e0cae0454330942ca4717ffa626a854c256925a63"
Oct 29 22:13:44.753279 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:13:44.748085    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76e8b0fde55763dad76d081e0cae0454330942ca4717ffa626a854c256925a63\": container with ID starting with 76e8b0fde55763dad76d081e0cae0454330942ca4717ffa626a854c256925a63 not found: ID does not exist" containerID="76e8b0fde55763dad76d081e0cae0454330942ca4717ffa626a854c256925a63"
Oct 29 22:13:44.753279 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:44.748125    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e8b0fde55763dad76d081e0cae0454330942ca4717ffa626a854c256925a63"} err="failed to get container status \"76e8b0fde55763dad76d081e0cae0454330942ca4717ffa626a854c256925a63\": rpc error: code = NotFound desc = could not find container \"76e8b0fde55763dad76d081e0cae0454330942ca4717ffa626a854c256925a63\": container with ID starting with 76e8b0fde55763dad76d081e0cae0454330942ca4717ffa626a854c256925a63 not found: ID does not exist"
Oct 29 22:13:44.829867 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:44.829830    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/674cc938-504c-4547-abb7-cc2ee413d6cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/674cc938-504c-4547-abb7-cc2ee413d6cf-metrics-certs\") pod \"674cc938-504c-4547-abb7-cc2ee413d6cf\" (UID: \"674cc938-504c-4547-abb7-cc2ee413d6cf\") "
Oct 29 22:13:44.838507 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:44.838473    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/674cc938-504c-4547-abb7-cc2ee413d6cf-kube-api-access-blfjw\" (UniqueName: \"kubernetes.io/projected/674cc938-504c-4547-abb7-cc2ee413d6cf-kube-api-access-blfjw\") pod \"674cc938-504c-4547-abb7-cc2ee413d6cf\" (UID: \"674cc938-504c-4547-abb7-cc2ee413d6cf\") "
Oct 29 22:13:44.844438 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:44.844408    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/674cc938-504c-4547-abb7-cc2ee413d6cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/674cc938-504c-4547-abb7-cc2ee413d6cf-maas-api-tls\") pod \"674cc938-504c-4547-abb7-cc2ee413d6cf\" (UID: \"674cc938-504c-4547-abb7-cc2ee413d6cf\") "
Oct 29 22:13:44.879365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:44.863861    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/674cc938-504c-4547-abb7-cc2ee413d6cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/674cc938-504c-4547-abb7-cc2ee413d6cf-openshift-service-ca\") pod \"674cc938-504c-4547-abb7-cc2ee413d6cf\" (UID: \"674cc938-504c-4547-abb7-cc2ee413d6cf\") "
Oct 29 22:13:44.977432 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:44.864724    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/674cc938-504c-4547-abb7-cc2ee413d6cf-openshift-service-ca" pod "674cc938-504c-4547-abb7-cc2ee413d6cf" (UID: "674cc938-504c-4547-abb7-cc2ee413d6cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:44.988308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:44.879874    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/674cc938-504c-4547-abb7-cc2ee413d6cf-maas-api-tls" pod "674cc938-504c-4547-abb7-cc2ee413d6cf" (UID: "674cc938-504c-4547-abb7-cc2ee413d6cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:13:44.990365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:44.979427    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/674cc938-504c-4547-abb7-cc2ee413d6cf-metrics-certs" pod "674cc938-504c-4547-abb7-cc2ee413d6cf" (UID: "674cc938-504c-4547-abb7-cc2ee413d6cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:13:45.051397 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:45.001595    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674cc938-504c-4547-abb7-cc2ee413d6cf-kube-api-access-blfjw" pod "674cc938-504c-4547-abb7-cc2ee413d6cf" (UID: "674cc938-504c-4547-abb7-cc2ee413d6cf"). InnerVolumeSpecName "kube-api-access-blfjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:45.051397 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:45.038931    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/674cc938-504c-4547-abb7-cc2ee413d6cf-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:45.051397 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:45.051398    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/674cc938-504c-4547-abb7-cc2ee413d6cf-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:45.153525 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:45.153442    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blfjw\" (UniqueName: \"kubernetes.io/projected/674cc938-504c-4547-abb7-cc2ee413d6cf-kube-api-access-blfjw\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:45.153525 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:45.153476    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/674cc938-504c-4547-abb7-cc2ee413d6cf-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:45.656890 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:45.656860    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44"
Oct 29 22:13:45.711487 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:45.711418    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44"]
Oct 29 22:13:45.722974 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:45.722951    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4e8583-5dbb8b949b-tmd44"]
Oct 29 22:13:47.059136 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:47.059108    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="674cc938-504c-4547-abb7-cc2ee413d6cf" path="/var/lib/kubelet/pods/674cc938-504c-4547-abb7-cc2ee413d6cf/volumes"
Oct 29 22:13:47.234347 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:47.234311    4962 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:13:47.234508 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:47.234376    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p" podUID="13d62170-e1c5-4478-886e-1b7b7f20d9e0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:13:48.055264 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:48.055230    4962 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:13:48.055454 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:48.055316    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg" podUID="be60e8e6-3cc5-45f2-b8f0-0d55d04e490a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:13:48.266401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:48.266365    4962 patch_prober.go:29] interesting pod/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:13:48.266770 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:48.266422    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv" podUID="654ebdbf-c9c1-4571-91b1-465aa5840051" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:13:49.807603 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.807564    4962 generic.go:356] "Generic (PLEG): container finished" podID="654ebdbf-c9c1-4571-91b1-465aa5840051" containerID="f686bd197707f6a55bf13ee343a2b4fb3d69320293fe1ba76fb230f5b7f6159c" exitCode=0
Oct 29 22:13:49.808102 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.807743    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv" event={"ID":"654ebdbf-c9c1-4571-91b1-465aa5840051","Type":"ContainerDied","Data":"f686bd197707f6a55bf13ee343a2b4fb3d69320293fe1ba76fb230f5b7f6159c"}
Oct 29 22:13:49.814698 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.814671    4962 generic.go:356] "Generic (PLEG): container finished" podID="13d62170-e1c5-4478-886e-1b7b7f20d9e0" containerID="8c351b4f6ebf661af34efaf64ed4ae5a99c86b78b241f60863c7c73b89a2a906" exitCode=0
Oct 29 22:13:49.814829 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.814755    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p" event={"ID":"13d62170-e1c5-4478-886e-1b7b7f20d9e0","Type":"ContainerDied","Data":"8c351b4f6ebf661af34efaf64ed4ae5a99c86b78b241f60863c7c73b89a2a906"}
Oct 29 22:13:49.815438 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.814827    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p" event={"ID":"13d62170-e1c5-4478-886e-1b7b7f20d9e0","Type":"ContainerDied","Data":"b16e3bbbb558ad404267506f499c472f0f740a4ac15573ad556561ccef20abaf"}
Oct 29 22:13:49.815520 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.815485    4962 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b16e3bbbb558ad404267506f499c472f0f740a4ac15573ad556561ccef20abaf"
Oct 29 22:13:49.816622 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.816602    4962 generic.go:356] "Generic (PLEG): container finished" podID="be60e8e6-3cc5-45f2-b8f0-0d55d04e490a" containerID="601e860facd433df5ae34264a45424e57f7390f9743d15dcbdc7d7bdd7fd9014" exitCode=0
Oct 29 22:13:49.816701 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.816630    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg" event={"ID":"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a","Type":"ContainerDied","Data":"601e860facd433df5ae34264a45424e57f7390f9743d15dcbdc7d7bdd7fd9014"}
Oct 29 22:13:49.850766 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.850742    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:13:49.855954 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.855931    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:13:49.957347 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.957274    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-credential-socket\") pod \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") "
Oct 29 22:13:49.957347 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.957340    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-envoy\") pod \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") "
Oct 29 22:13:49.958281 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.957422    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-credential-socket\") pod \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") "
Oct 29 22:13:49.958281 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.957468    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-kube-api-access-lfh9g\" (UniqueName: \"kubernetes.io/projected/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-kube-api-access-lfh9g\") pod \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") "
Oct 29 22:13:49.958281 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.957538    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13d62170-e1c5-4478-886e-1b7b7f20d9e0-kube-api-access-8lj5s\" (UniqueName: \"kubernetes.io/projected/13d62170-e1c5-4478-886e-1b7b7f20d9e0-kube-api-access-8lj5s\") pod \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") "
Oct 29 22:13:49.958281 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.957581    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-token\" (UniqueName: \"kubernetes.io/projected/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-token\") pod \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") "
Oct 29 22:13:49.958281 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.957640    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-podinfo\") pod \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") "
Oct 29 22:13:49.958281 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.957741    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-data\") pod \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") "
Oct 29 22:13:49.958281 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.957817    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-data\") pod \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") "
Oct 29 22:13:49.958281 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.957866    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-workload-certs\") pod \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") "
Oct 29 22:13:49.958281 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.957919    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-workload-socket\") pod \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") "
Oct 29 22:13:49.958281 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.957959    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istiod-ca-cert\") pod \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") "
Oct 29 22:13:49.958281 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.958008    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-podinfo\") pod \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") "
Oct 29 22:13:49.958281 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.958040    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-token\" (UniqueName: \"kubernetes.io/projected/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-token\") pod \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\" (UID: \"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a\") "
Oct 29 22:13:49.958281 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.958091    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-workload-certs\") pod \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") "
Oct 29 22:13:49.958281 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.958137    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-workload-socket\") pod \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") "
Oct 29 22:13:49.958281 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.958173    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istiod-ca-cert\") pod \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") "
Oct 29 22:13:49.958281 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.958201    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-envoy\") pod \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\" (UID: \"13d62170-e1c5-4478-886e-1b7b7f20d9e0\") "
Oct 29 22:13:49.959583 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.959554    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-data" pod "13d62170-e1c5-4478-886e-1b7b7f20d9e0" (UID: "13d62170-e1c5-4478-886e-1b7b7f20d9e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:49.960391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.959803    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-workload-socket" pod "be60e8e6-3cc5-45f2-b8f0-0d55d04e490a" (UID: "be60e8e6-3cc5-45f2-b8f0-0d55d04e490a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:49.960391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.960031    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-workload-socket" pod "13d62170-e1c5-4478-886e-1b7b7f20d9e0" (UID: "13d62170-e1c5-4478-886e-1b7b7f20d9e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:49.960391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.960043    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-workload-certs" pod "be60e8e6-3cc5-45f2-b8f0-0d55d04e490a" (UID: "be60e8e6-3cc5-45f2-b8f0-0d55d04e490a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:49.960391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.960225    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-credential-socket" pod "13d62170-e1c5-4478-886e-1b7b7f20d9e0" (UID: "13d62170-e1c5-4478-886e-1b7b7f20d9e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:49.960391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.960323    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istiod-ca-cert" pod "13d62170-e1c5-4478-886e-1b7b7f20d9e0" (UID: "13d62170-e1c5-4478-886e-1b7b7f20d9e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:49.961027 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.960766    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istiod-ca-cert" pod "be60e8e6-3cc5-45f2-b8f0-0d55d04e490a" (UID: "be60e8e6-3cc5-45f2-b8f0-0d55d04e490a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:49.961027 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.960998    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-data" pod "be60e8e6-3cc5-45f2-b8f0-0d55d04e490a" (UID: "be60e8e6-3cc5-45f2-b8f0-0d55d04e490a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:49.961912 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.961865    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-workload-certs" pod "13d62170-e1c5-4478-886e-1b7b7f20d9e0" (UID: "13d62170-e1c5-4478-886e-1b7b7f20d9e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:49.962200 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.962177    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-credential-socket" pod "be60e8e6-3cc5-45f2-b8f0-0d55d04e490a" (UID: "be60e8e6-3cc5-45f2-b8f0-0d55d04e490a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:49.965254 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.965214    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-token" pod "13d62170-e1c5-4478-886e-1b7b7f20d9e0" (UID: "13d62170-e1c5-4478-886e-1b7b7f20d9e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:49.965905 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.965439    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-podinfo" pod "be60e8e6-3cc5-45f2-b8f0-0d55d04e490a" (UID: "be60e8e6-3cc5-45f2-b8f0-0d55d04e490a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:13:49.965905 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.965494    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d62170-e1c5-4478-886e-1b7b7f20d9e0-kube-api-access-8lj5s" pod "13d62170-e1c5-4478-886e-1b7b7f20d9e0" (UID: "13d62170-e1c5-4478-886e-1b7b7f20d9e0"). InnerVolumeSpecName "kube-api-access-8lj5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:49.966109 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.966057    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-envoy" pod "be60e8e6-3cc5-45f2-b8f0-0d55d04e490a" (UID: "be60e8e6-3cc5-45f2-b8f0-0d55d04e490a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:49.966384 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.966356    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-envoy" pod "13d62170-e1c5-4478-886e-1b7b7f20d9e0" (UID: "13d62170-e1c5-4478-886e-1b7b7f20d9e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:49.966704 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.966672    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-kube-api-access-lfh9g" pod "be60e8e6-3cc5-45f2-b8f0-0d55d04e490a" (UID: "be60e8e6-3cc5-45f2-b8f0-0d55d04e490a"). InnerVolumeSpecName "kube-api-access-lfh9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:49.967073 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.967048    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-token" pod "be60e8e6-3cc5-45f2-b8f0-0d55d04e490a" (UID: "be60e8e6-3cc5-45f2-b8f0-0d55d04e490a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:49.967228 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:49.967206    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-podinfo" pod "13d62170-e1c5-4478-886e-1b7b7f20d9e0" (UID: "13d62170-e1c5-4478-886e-1b7b7f20d9e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:13:50.061178 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.061106    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.061178 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.061137    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfh9g\" (UniqueName: \"kubernetes.io/projected/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-kube-api-access-lfh9g\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.061178 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.061151    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lj5s\" (UniqueName: \"kubernetes.io/projected/13d62170-e1c5-4478-886e-1b7b7f20d9e0-kube-api-access-8lj5s\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.061178 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.061173    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.061548 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.061188    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.061548 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.061199    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.061548 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.061210    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.061548 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.061222    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.061548 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.061372    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.061548 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.061388    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.061548 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.061403    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.061548 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.061423    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.061548 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.061436    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.061548 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.061449    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.061548 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.061463    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.061548 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.061522    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d62170-e1c5-4478-886e-1b7b7f20d9e0-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.062019 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.061545    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.062019 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.061624    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.223270 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.223239    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:13:50.264819 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.264778    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/654ebdbf-c9c1-4571-91b1-465aa5840051-kube-api-access-96q6j\" (UniqueName: \"kubernetes.io/projected/654ebdbf-c9c1-4571-91b1-465aa5840051-kube-api-access-96q6j\") pod \"654ebdbf-c9c1-4571-91b1-465aa5840051\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") "
Oct 29 22:13:50.264951 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.264844    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-workload-certs\") pod \"654ebdbf-c9c1-4571-91b1-465aa5840051\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") "
Oct 29 22:13:50.265016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.264980    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-envoy\") pod \"654ebdbf-c9c1-4571-91b1-465aa5840051\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") "
Oct 29 22:13:50.265159 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.265130    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-token\" (UniqueName: \"kubernetes.io/projected/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-token\") pod \"654ebdbf-c9c1-4571-91b1-465aa5840051\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") "
Oct 29 22:13:50.265230 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.265159    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-workload-certs" pod "654ebdbf-c9c1-4571-91b1-465aa5840051" (UID: "654ebdbf-c9c1-4571-91b1-465aa5840051"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.265230 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.265213    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-credential-socket\") pod \"654ebdbf-c9c1-4571-91b1-465aa5840051\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") "
Oct 29 22:13:50.265418 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.265376    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/654ebdbf-c9c1-4571-91b1-465aa5840051-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/654ebdbf-c9c1-4571-91b1-465aa5840051-istiod-ca-cert\") pod \"654ebdbf-c9c1-4571-91b1-465aa5840051\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") "
Oct 29 22:13:50.265418 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.265418    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-podinfo\") pod \"654ebdbf-c9c1-4571-91b1-465aa5840051\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") "
Oct 29 22:13:50.265647 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.265511    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-data\") pod \"654ebdbf-c9c1-4571-91b1-465aa5840051\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") "
Oct 29 22:13:50.265647 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.265612    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-workload-socket\") pod \"654ebdbf-c9c1-4571-91b1-465aa5840051\" (UID: \"654ebdbf-c9c1-4571-91b1-465aa5840051\") "
Oct 29 22:13:50.265941 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.265507    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-credential-socket" pod "654ebdbf-c9c1-4571-91b1-465aa5840051" (UID: "654ebdbf-c9c1-4571-91b1-465aa5840051"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.266027 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.265875    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/654ebdbf-c9c1-4571-91b1-465aa5840051-istiod-ca-cert" pod "654ebdbf-c9c1-4571-91b1-465aa5840051" (UID: "654ebdbf-c9c1-4571-91b1-465aa5840051"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:13:50.266144 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.266120    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-data" pod "654ebdbf-c9c1-4571-91b1-465aa5840051" (UID: "654ebdbf-c9c1-4571-91b1-465aa5840051"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.266213 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.266152    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-workload-socket" pod "654ebdbf-c9c1-4571-91b1-465aa5840051" (UID: "654ebdbf-c9c1-4571-91b1-465aa5840051"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.266558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.266536    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.266614 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.266565    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/654ebdbf-c9c1-4571-91b1-465aa5840051-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.266614 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.266580    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.266614 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.266592    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.266614 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.266605    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.267446 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.267423    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-envoy" pod "654ebdbf-c9c1-4571-91b1-465aa5840051" (UID: "654ebdbf-c9c1-4571-91b1-465aa5840051"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:50.267515 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.267455    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/654ebdbf-c9c1-4571-91b1-465aa5840051-kube-api-access-96q6j" pod "654ebdbf-c9c1-4571-91b1-465aa5840051" (UID: "654ebdbf-c9c1-4571-91b1-465aa5840051"). InnerVolumeSpecName "kube-api-access-96q6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:50.267567 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.267543    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-token" pod "654ebdbf-c9c1-4571-91b1-465aa5840051" (UID: "654ebdbf-c9c1-4571-91b1-465aa5840051"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:50.267999 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.267979    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-podinfo" pod "654ebdbf-c9c1-4571-91b1-465aa5840051" (UID: "654ebdbf-c9c1-4571-91b1-465aa5840051"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:13:50.368042 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.367959    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.368042 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.367992    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.368042 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.368001    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/654ebdbf-c9c1-4571-91b1-465aa5840051-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.368042 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.368010    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96q6j\" (UniqueName: \"kubernetes.io/projected/654ebdbf-c9c1-4571-91b1-465aa5840051-kube-api-access-96q6j\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:50.826176 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.826144    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"
Oct 29 22:13:50.826176 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.826166    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv" event={"ID":"654ebdbf-c9c1-4571-91b1-465aa5840051","Type":"ContainerDied","Data":"293dd24f88267d659167f093fc110637d1a62bcc6278eb5d876d258b84aea2d5"}
Oct 29 22:13:50.826711 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.826229    4962 scope.go:122] "RemoveContainer" containerID="f686bd197707f6a55bf13ee343a2b4fb3d69320293fe1ba76fb230f5b7f6159c"
Oct 29 22:13:50.828001 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.827981    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"
Oct 29 22:13:50.828001 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.827991    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"
Oct 29 22:13:50.828194 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.828024    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg" event={"ID":"be60e8e6-3cc5-45f2-b8f0-0d55d04e490a","Type":"ContainerDied","Data":"eb6d4320a3dcda6889c700762319ce42f8036185b3ea674afa26b925e8bc3481"}
Oct 29 22:13:50.840016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.839998    4962 scope.go:122] "RemoveContainer" containerID="601e860facd433df5ae34264a45424e57f7390f9743d15dcbdc7d7bdd7fd9014"
Oct 29 22:13:50.903912 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.903880    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"]
Oct 29 22:13:50.931173 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.931150    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-s4wjg"]
Oct 29 22:13:50.966101 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.966072    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"]
Oct 29 22:13:50.971774 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:50.971753    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-mwg9p"]
Oct 29 22:13:51.006724 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.006700    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"]
Oct 29 22:13:51.011019 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.010998    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-48c52b-openshift-default-6d88cd8ddc-xh2bv"]
Oct 29 22:13:51.059645 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.059616    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13d62170-e1c5-4478-886e-1b7b7f20d9e0" path="/var/lib/kubelet/pods/13d62170-e1c5-4478-886e-1b7b7f20d9e0/volumes"
Oct 29 22:13:51.060270 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.060249    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="654ebdbf-c9c1-4571-91b1-465aa5840051" path="/var/lib/kubelet/pods/654ebdbf-c9c1-4571-91b1-465aa5840051/volumes"
Oct 29 22:13:51.060827 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.060799    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be60e8e6-3cc5-45f2-b8f0-0d55d04e490a" path="/var/lib/kubelet/pods/be60e8e6-3cc5-45f2-b8f0-0d55d04e490a/volumes"
Oct 29 22:13:51.610341 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.610282    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" containerID="cri-o://a77d84c5e175f77734846f3530da23ce5f1c6b6f6c657158598344e048a8a84c" gracePeriod=45
Oct 29 22:13:51.615681 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.615649    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" containerID="cri-o://150138e807a2048dcc0dc7554b609bcd44307c79135a8e9467f8320806e82566" gracePeriod=45
Oct 29 22:13:51.618029 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.617971    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main" containerID="cri-o://7dffb80a37b687e99dfa2404188d26dcf375bc193b0616d63e0f87c6fcb8810d" gracePeriod=45
Oct 29 22:13:51.665355 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.665326    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": dial tcp 10.132.0.86:8000: connect: connection refused"
Oct 29 22:13:51.691266 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.691233    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": dial tcp 10.132.0.80:8000: connect: connection refused"
Oct 29 22:13:51.839555 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.839527    4962 generic.go:356] "Generic (PLEG): container finished" podID="2883ab31-9399-4c28-af6a-9c66fd043396" containerID="7dffb80a37b687e99dfa2404188d26dcf375bc193b0616d63e0f87c6fcb8810d" exitCode=0
Oct 29 22:13:51.839929 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.839600    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" event={"ID":"2883ab31-9399-4c28-af6a-9c66fd043396","Type":"ContainerDied","Data":"7dffb80a37b687e99dfa2404188d26dcf375bc193b0616d63e0f87c6fcb8810d"}
Oct 29 22:13:51.841517 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.841498    4962 generic.go:356] "Generic (PLEG): container finished" podID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerID="150138e807a2048dcc0dc7554b609bcd44307c79135a8e9467f8320806e82566" exitCode=0
Oct 29 22:13:51.841621 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.841580    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" event={"ID":"e548eb7e-ffd6-48b0-8358-30a081d3cf4b","Type":"ContainerDied","Data":"150138e807a2048dcc0dc7554b609bcd44307c79135a8e9467f8320806e82566"}
Oct 29 22:13:51.844537 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.844514    4962 generic.go:356] "Generic (PLEG): container finished" podID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerID="a77d84c5e175f77734846f3530da23ce5f1c6b6f6c657158598344e048a8a84c" exitCode=0
Oct 29 22:13:51.844712 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.844591    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" event={"ID":"b51ab222-55f7-4fa1-b39d-0c48a06a7173","Type":"ContainerDied","Data":"a77d84c5e175f77734846f3530da23ce5f1c6b6f6c657158598344e048a8a84c"}
Oct 29 22:13:51.979718 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.969579    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 22:13:51.984076 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.984050    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 22:13:51.986283 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.986263    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 22:13:51.995442 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.995421    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-model-cache\") pod \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\" (UID: \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\") "
Oct 29 22:13:51.995552 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.995531    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b51ab222-55f7-4fa1-b39d-0c48a06a7173-tls-certs\" (UniqueName: \"kubernetes.io/secret/b51ab222-55f7-4fa1-b39d-0c48a06a7173-tls-certs\") pod \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\" (UID: \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\") "
Oct 29 22:13:51.995627 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.995590    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-tmp-dir\") pod \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\" (UID: \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\") "
Oct 29 22:13:51.995681 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.995632    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-home\" (UniqueName: \"kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-home\") pod \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\" (UID: \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\") "
Oct 29 22:13:51.995731 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.995707    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-model-cache" pod "b51ab222-55f7-4fa1-b39d-0c48a06a7173" (UID: "b51ab222-55f7-4fa1-b39d-0c48a06a7173"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:51.995877 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.995855    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-dshm\") pod \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\" (UID: \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\") "
Oct 29 22:13:51.995941 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.995880    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-tmp-dir" pod "b51ab222-55f7-4fa1-b39d-0c48a06a7173" (UID: "b51ab222-55f7-4fa1-b39d-0c48a06a7173"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:51.995941 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.995896    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-home" pod "b51ab222-55f7-4fa1-b39d-0c48a06a7173" (UID: "b51ab222-55f7-4fa1-b39d-0c48a06a7173"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:51.996043 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.995952    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b51ab222-55f7-4fa1-b39d-0c48a06a7173-kube-api-access-fc8ts\" (UniqueName: \"kubernetes.io/projected/b51ab222-55f7-4fa1-b39d-0c48a06a7173-kube-api-access-fc8ts\") pod \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\" (UID: \"b51ab222-55f7-4fa1-b39d-0c48a06a7173\") "
Oct 29 22:13:51.996788 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.996766    4962 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-model-cache\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:51.996924 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.996793    4962 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-tmp-dir\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:51.996924 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.996817    4962 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-home\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:51.998169 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.998137    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b51ab222-55f7-4fa1-b39d-0c48a06a7173-tls-certs" pod "b51ab222-55f7-4fa1-b39d-0c48a06a7173" (UID: "b51ab222-55f7-4fa1-b39d-0c48a06a7173"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:13:51.998169 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.998152    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-dshm" pod "b51ab222-55f7-4fa1-b39d-0c48a06a7173" (UID: "b51ab222-55f7-4fa1-b39d-0c48a06a7173"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:51.998769 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:51.998747    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51ab222-55f7-4fa1-b39d-0c48a06a7173-kube-api-access-fc8ts" pod "b51ab222-55f7-4fa1-b39d-0c48a06a7173" (UID: "b51ab222-55f7-4fa1-b39d-0c48a06a7173"). InnerVolumeSpecName "kube-api-access-fc8ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:52.099740 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.098248    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-dshm\") pod \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\" (UID: \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\") "
Oct 29 22:13:52.099740 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.098351    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2883ab31-9399-4c28-af6a-9c66fd043396-kube-api-access-k9gqk\" (UniqueName: \"kubernetes.io/projected/2883ab31-9399-4c28-af6a-9c66fd043396-kube-api-access-k9gqk\") pod \"2883ab31-9399-4c28-af6a-9c66fd043396\" (UID: \"2883ab31-9399-4c28-af6a-9c66fd043396\") "
Oct 29 22:13:52.099740 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.098473    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-home\" (UniqueName: \"kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-home\") pod \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\" (UID: \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\") "
Oct 29 22:13:52.099740 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.098520    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-tmp-dir\") pod \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\" (UID: \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\") "
Oct 29 22:13:52.099740 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.098574    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-tmp-dir\") pod \"2883ab31-9399-4c28-af6a-9c66fd043396\" (UID: \"2883ab31-9399-4c28-af6a-9c66fd043396\") "
Oct 29 22:13:52.099740 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.098744    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-dshm\") pod \"2883ab31-9399-4c28-af6a-9c66fd043396\" (UID: \"2883ab31-9399-4c28-af6a-9c66fd043396\") "
Oct 29 22:13:52.099740 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.098821    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-model-cache\") pod \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\" (UID: \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\") "
Oct 29 22:13:52.099740 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.098875    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-model-cache\") pod \"2883ab31-9399-4c28-af6a-9c66fd043396\" (UID: \"2883ab31-9399-4c28-af6a-9c66fd043396\") "
Oct 29 22:13:52.099740 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.098915    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-home\" (UniqueName: \"kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-home\") pod \"2883ab31-9399-4c28-af6a-9c66fd043396\" (UID: \"2883ab31-9399-4c28-af6a-9c66fd043396\") "
Oct 29 22:13:52.099740 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.098994    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2883ab31-9399-4c28-af6a-9c66fd043396-tls-certs\" (UniqueName: \"kubernetes.io/secret/2883ab31-9399-4c28-af6a-9c66fd043396-tls-certs\") pod \"2883ab31-9399-4c28-af6a-9c66fd043396\" (UID: \"2883ab31-9399-4c28-af6a-9c66fd043396\") "
Oct 29 22:13:52.099740 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.099102    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-kube-api-access-vwq2q\" (UniqueName: \"kubernetes.io/projected/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-kube-api-access-vwq2q\") pod \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\" (UID: \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\") "
Oct 29 22:13:52.099740 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.099277    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-tls-certs\" (UniqueName: \"kubernetes.io/secret/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-tls-certs\") pod \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\" (UID: \"e548eb7e-ffd6-48b0-8358-30a081d3cf4b\") "
Oct 29 22:13:52.102179 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.102148    4962 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b51ab222-55f7-4fa1-b39d-0c48a06a7173-tls-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:52.102324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.102184    4962 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b51ab222-55f7-4fa1-b39d-0c48a06a7173-dshm\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:52.102324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.102197    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc8ts\" (UniqueName: \"kubernetes.io/projected/b51ab222-55f7-4fa1-b39d-0c48a06a7173-kube-api-access-fc8ts\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:52.102665 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.102640    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-model-cache" pod "e548eb7e-ffd6-48b0-8358-30a081d3cf4b" (UID: "e548eb7e-ffd6-48b0-8358-30a081d3cf4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:52.102892 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.102869    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-model-cache" pod "2883ab31-9399-4c28-af6a-9c66fd043396" (UID: "2883ab31-9399-4c28-af6a-9c66fd043396"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:52.103103 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.103080    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-home" pod "2883ab31-9399-4c28-af6a-9c66fd043396" (UID: "2883ab31-9399-4c28-af6a-9c66fd043396"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:52.105028 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.105001    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-tmp-dir" pod "e548eb7e-ffd6-48b0-8358-30a081d3cf4b" (UID: "e548eb7e-ffd6-48b0-8358-30a081d3cf4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:52.105841 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.105813    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-home" pod "e548eb7e-ffd6-48b0-8358-30a081d3cf4b" (UID: "e548eb7e-ffd6-48b0-8358-30a081d3cf4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:52.106213 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.106192    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-dshm" pod "2883ab31-9399-4c28-af6a-9c66fd043396" (UID: "2883ab31-9399-4c28-af6a-9c66fd043396"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:52.106354 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.106206    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-tmp-dir" pod "2883ab31-9399-4c28-af6a-9c66fd043396" (UID: "2883ab31-9399-4c28-af6a-9c66fd043396"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:52.108697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.108668    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2883ab31-9399-4c28-af6a-9c66fd043396-kube-api-access-k9gqk" pod "2883ab31-9399-4c28-af6a-9c66fd043396" (UID: "2883ab31-9399-4c28-af6a-9c66fd043396"). InnerVolumeSpecName "kube-api-access-k9gqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:52.109634 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.109613    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-tls-certs" pod "e548eb7e-ffd6-48b0-8358-30a081d3cf4b" (UID: "e548eb7e-ffd6-48b0-8358-30a081d3cf4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:13:52.109724 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.109635    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2883ab31-9399-4c28-af6a-9c66fd043396-tls-certs" pod "2883ab31-9399-4c28-af6a-9c66fd043396" (UID: "2883ab31-9399-4c28-af6a-9c66fd043396"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:13:52.109724 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.109680    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-kube-api-access-vwq2q" pod "e548eb7e-ffd6-48b0-8358-30a081d3cf4b" (UID: "e548eb7e-ffd6-48b0-8358-30a081d3cf4b"). InnerVolumeSpecName "kube-api-access-vwq2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:13:52.111098 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.111079    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-dshm" pod "e548eb7e-ffd6-48b0-8358-30a081d3cf4b" (UID: "e548eb7e-ffd6-48b0-8358-30a081d3cf4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:13:52.207383 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.207061    4962 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-dshm\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:52.207383 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.207111    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9gqk\" (UniqueName: \"kubernetes.io/projected/2883ab31-9399-4c28-af6a-9c66fd043396-kube-api-access-k9gqk\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:52.207383 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.207130    4962 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-home\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:52.207383 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.207146    4962 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-tmp-dir\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:52.207383 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.207160    4962 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-tmp-dir\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:52.207383 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.207171    4962 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-dshm\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:52.207383 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.207183    4962 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-model-cache\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:52.207383 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.207195    4962 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-model-cache\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:52.207383 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.207208    4962 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2883ab31-9399-4c28-af6a-9c66fd043396-home\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:52.207383 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.207219    4962 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2883ab31-9399-4c28-af6a-9c66fd043396-tls-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:52.207383 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.207232    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwq2q\" (UniqueName: \"kubernetes.io/projected/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-kube-api-access-vwq2q\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:52.207383 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.207247    4962 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e548eb7e-ffd6-48b0-8358-30a081d3cf4b-tls-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:13:52.866542 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.866514    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"
Oct 29 22:13:52.867528 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.866514    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h" event={"ID":"2883ab31-9399-4c28-af6a-9c66fd043396","Type":"ContainerDied","Data":"aaba27141d62906b0c54449a27ebb0ae813698022017463cf06774695ed14327"}
Oct 29 22:13:52.867666 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.867554    4962 scope.go:122] "RemoveContainer" containerID="7dffb80a37b687e99dfa2404188d26dcf375bc193b0616d63e0f87c6fcb8810d"
Oct 29 22:13:52.868075 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.868051    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"
Oct 29 22:13:52.868210 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.868074    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p" event={"ID":"e548eb7e-ffd6-48b0-8358-30a081d3cf4b","Type":"ContainerDied","Data":"7681049fcaa4b56fe30826133bb0f9ad2a71503198091eceaddeae13ce717579"}
Oct 29 22:13:52.869924 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.869905    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"
Oct 29 22:13:52.870050 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.869928    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c" event={"ID":"b51ab222-55f7-4fa1-b39d-0c48a06a7173","Type":"ContainerDied","Data":"df6e4611e40c409ebc5800fc2032a61ac9a66d86714b0b53f7e6081156280492"}
Oct 29 22:13:52.880624 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:52.880400    4962 scope.go:122] "RemoveContainer" containerID="150138e807a2048dcc0dc7554b609bcd44307c79135a8e9467f8320806e82566"
Oct 29 22:13:53.012778 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:53.008534    4962 scope.go:122] "RemoveContainer" containerID="a77d84c5e175f77734846f3530da23ce5f1c6b6f6c657158598344e048a8a84c"
Oct 29 22:13:53.029977 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:53.029945    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"]
Oct 29 22:13:53.035887 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:53.035856    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b31f3f/premium-w4-b31f3f-kserve-66f587688f-bdc4h"]
Oct 29 22:13:53.044832 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:53.044796    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"]
Oct 29 22:13:53.048479 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:53.048451    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b31f3f/distinct-2-w4-b31f3f-kserve-665f49cc95-7jf6p"]
Oct 29 22:13:53.089251 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:53.089211    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" path="/var/lib/kubelet/pods/2883ab31-9399-4c28-af6a-9c66fd043396/volumes"
Oct 29 22:13:53.090373 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:53.090346    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" path="/var/lib/kubelet/pods/e548eb7e-ffd6-48b0-8358-30a081d3cf4b/volumes"
Oct 29 22:13:53.161334 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:53.161303    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"]
Oct 29 22:13:53.177378 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:53.177345    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b31f3f/distinct-w4-b31f3f-kserve-d6bd54f5b-9d99c"]
Oct 29 22:13:53.295254 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:53.295221    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"]
Oct 29 22:13:53.295492 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:53.295450    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl" podUID="093f784b-0baf-4f14-984e-ec8003db30fb" containerName="istio-proxy" containerID="cri-o://97dd00635c0d257493e656a1cc325cac4e89fb9f39fead1fb6213c33bf7f7299" gracePeriod=30
Oct 29 22:13:53.592058 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:53.591932    4962 patch_prober.go:29] interesting pod/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:13:53.592417 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:53.592355    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl" podUID="093f784b-0baf-4f14-984e-ec8003db30fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:13:55.063970 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.063929    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" path="/var/lib/kubelet/pods/b51ab222-55f7-4fa1-b39d-0c48a06a7173/volumes"
Oct 29 22:13:55.131388 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.131359    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"]
Oct 29 22:13:55.132033 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132016    4962 cpu_manager.go:414] "containerMap: removing container" podUID="654ebdbf-c9c1-4571-91b1-465aa5840051" containerName="istio-proxy"
Oct 29 22:13:55.132033 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132034    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="654ebdbf-c9c1-4571-91b1-465aa5840051" containerName="istio-proxy"
Oct 29 22:13:55.132154 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132049    4962 cpu_manager.go:414] "containerMap: removing container" podUID="68dfa23b-de59-4864-a08e-7ab7185792a8" containerName="payload-pre-processing"
Oct 29 22:13:55.132154 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132054    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68dfa23b-de59-4864-a08e-7ab7185792a8" containerName="payload-pre-processing"
Oct 29 22:13:55.132154 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132062    4962 cpu_manager.go:414] "containerMap: removing container" podUID="5e577b68-1188-430f-9930-eb43bc82dd07" containerName="maas-api"
Oct 29 22:13:55.132154 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132067    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e577b68-1188-430f-9930-eb43bc82dd07" containerName="maas-api"
Oct 29 22:13:55.132154 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132083    4962 cpu_manager.go:414] "containerMap: removing container" podUID="4c2c8ef0-de16-4e8c-b524-21e8151dc910" containerName="payload-pre-processing"
Oct 29 22:13:55.132154 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132088    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c2c8ef0-de16-4e8c-b524-21e8151dc910" containerName="payload-pre-processing"
Oct 29 22:13:55.132154 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132098    4962 cpu_manager.go:414] "containerMap: removing container" podUID="13d62170-e1c5-4478-886e-1b7b7f20d9e0" containerName="istio-proxy"
Oct 29 22:13:55.132154 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132103    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13d62170-e1c5-4478-886e-1b7b7f20d9e0" containerName="istio-proxy"
Oct 29 22:13:55.132154 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132110    4962 cpu_manager.go:414] "containerMap: removing container" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main"
Oct 29 22:13:55.132154 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132115    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main"
Oct 29 22:13:55.132154 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132123    4962 cpu_manager.go:414] "containerMap: removing container" podUID="df517504-590f-4f2b-8ab1-5e92e1f1936c" containerName="maas-api"
Oct 29 22:13:55.132154 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132128    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df517504-590f-4f2b-8ab1-5e92e1f1936c" containerName="maas-api"
Oct 29 22:13:55.132154 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132144    4962 cpu_manager.go:414] "containerMap: removing container" podUID="0897e113-b10f-460b-9b0f-17ce9daea2c7" containerName="payload-processing"
Oct 29 22:13:55.132154 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132148    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0897e113-b10f-460b-9b0f-17ce9daea2c7" containerName="payload-processing"
Oct 29 22:13:55.132154 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132154    4962 cpu_manager.go:414] "containerMap: removing container" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main"
Oct 29 22:13:55.132154 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132159    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main"
Oct 29 22:13:55.132678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132169    4962 cpu_manager.go:414] "containerMap: removing container" podUID="be60e8e6-3cc5-45f2-b8f0-0d55d04e490a" containerName="istio-proxy"
Oct 29 22:13:55.132678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132174    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be60e8e6-3cc5-45f2-b8f0-0d55d04e490a" containerName="istio-proxy"
Oct 29 22:13:55.132678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132183    4962 cpu_manager.go:414] "containerMap: removing container" podUID="faea6cb2-afaf-49a8-81ab-bb9c3f699e35" containerName="istio-proxy"
Oct 29 22:13:55.132678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132188    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faea6cb2-afaf-49a8-81ab-bb9c3f699e35" containerName="istio-proxy"
Oct 29 22:13:55.132678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132196    4962 cpu_manager.go:414] "containerMap: removing container" podUID="faea6cb2-afaf-49a8-81ab-bb9c3f699e35" containerName="istio-proxy"
Oct 29 22:13:55.132678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132200    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faea6cb2-afaf-49a8-81ab-bb9c3f699e35" containerName="istio-proxy"
Oct 29 22:13:55.132678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132211    4962 cpu_manager.go:414] "containerMap: removing container" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main"
Oct 29 22:13:55.132678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132215    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main"
Oct 29 22:13:55.132678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132223    4962 cpu_manager.go:414] "containerMap: removing container" podUID="674cc938-504c-4547-abb7-cc2ee413d6cf" containerName="maas-api"
Oct 29 22:13:55.132678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132227    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="674cc938-504c-4547-abb7-cc2ee413d6cf" containerName="maas-api"
Oct 29 22:13:55.132678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132347    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="e548eb7e-ffd6-48b0-8358-30a081d3cf4b" containerName="main"
Oct 29 22:13:55.132678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132355    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="df517504-590f-4f2b-8ab1-5e92e1f1936c" containerName="maas-api"
Oct 29 22:13:55.132678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132363    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="0897e113-b10f-460b-9b0f-17ce9daea2c7" containerName="payload-processing"
Oct 29 22:13:55.132678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132371    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="b51ab222-55f7-4fa1-b39d-0c48a06a7173" containerName="main"
Oct 29 22:13:55.132678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132379    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="be60e8e6-3cc5-45f2-b8f0-0d55d04e490a" containerName="istio-proxy"
Oct 29 22:13:55.132678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132387    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="faea6cb2-afaf-49a8-81ab-bb9c3f699e35" containerName="istio-proxy"
Oct 29 22:13:55.132678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132393    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="faea6cb2-afaf-49a8-81ab-bb9c3f699e35" containerName="istio-proxy"
Oct 29 22:13:55.132678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132399    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="2883ab31-9399-4c28-af6a-9c66fd043396" containerName="main"
Oct 29 22:13:55.132678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132404    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="674cc938-504c-4547-abb7-cc2ee413d6cf" containerName="maas-api"
Oct 29 22:13:55.132678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132413    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="654ebdbf-c9c1-4571-91b1-465aa5840051" containerName="istio-proxy"
Oct 29 22:13:55.132678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132419    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="68dfa23b-de59-4864-a08e-7ab7185792a8" containerName="payload-pre-processing"
Oct 29 22:13:55.132678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132425    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e577b68-1188-430f-9930-eb43bc82dd07" containerName="maas-api"
Oct 29 22:13:55.132678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132434    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c2c8ef0-de16-4e8c-b524-21e8151dc910" containerName="payload-pre-processing"
Oct 29 22:13:55.132678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.132441    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="13d62170-e1c5-4478-886e-1b7b7f20d9e0" containerName="istio-proxy"
Oct 29 22:13:55.138161 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.138142    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.169901 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.169873    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"]
Oct 29 22:13:55.293328 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.292407    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-istio-envoy\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.293328 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.292491    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-istio-data\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.293328 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.292577    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71174a8c-e386-4670-942e-3bc750987080-istio-podinfo\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.293328 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.292639    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-workload-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.293328 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.292693    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71174a8c-e386-4670-942e-3bc750987080-istio-token\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.293328 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.292769    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp8ht\" (UniqueName: \"kubernetes.io/projected/71174a8c-e386-4670-942e-3bc750987080-kube-api-access-sp8ht\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.293328 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.292821    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-workload-certs\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.293328 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.293154    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71174a8c-e386-4670-942e-3bc750987080-istiod-ca-cert\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.333775 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.328399    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-credential-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.430544 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.430512    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-credential-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.430737 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.430612    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-istio-envoy\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.430737 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.430668    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-istio-data\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.430737 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.430732    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71174a8c-e386-4670-942e-3bc750987080-istio-podinfo\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.430918 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.430785    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-workload-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.430918 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.430838    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71174a8c-e386-4670-942e-3bc750987080-istio-token\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.430918 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.430911    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp8ht\" (UniqueName: \"kubernetes.io/projected/71174a8c-e386-4670-942e-3bc750987080-kube-api-access-sp8ht\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.431081 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.430954    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-workload-certs\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.431081 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.430974    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-credential-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.431081 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.431066    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71174a8c-e386-4670-942e-3bc750987080-istiod-ca-cert\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.431248 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.431154    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-workload-socket\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.431341 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.431279    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-workload-certs\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.431713 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.431687    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-istio-data\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.431900 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.431876    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71174a8c-e386-4670-942e-3bc750987080-istiod-ca-cert\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.433143 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.433118    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-istio-envoy\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.433478 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.433461    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71174a8c-e386-4670-942e-3bc750987080-istio-podinfo\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.453034 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.453015    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71174a8c-e386-4670-942e-3bc750987080-istio-token\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.453555 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.453521    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp8ht\" (UniqueName: \"kubernetes.io/projected/71174a8c-e386-4670-942e-3bc750987080-kube-api-access-sp8ht\") pod \"e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") " pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.753163 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.753135    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:55.960138 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.960110    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr"]
Oct 29 22:13:55.992737 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:55.992700    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq"]
Oct 29 22:13:56.081247 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:13:56.081209    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71174a8c_e386_4670_942e_3bc750987080.slice/crio-b8f0649fadcbfee3386c89a4f15bdf65ba2aee1f4532002785c048ddcb663838 WatchSource:0}: Error finding container b8f0649fadcbfee3386c89a4f15bdf65ba2aee1f4532002785c048ddcb663838: Status 404 returned error can't find the container with id b8f0649fadcbfee3386c89a4f15bdf65ba2aee1f4532002785c048ddcb663838
Oct 29 22:13:56.082076 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:56.082001    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"]
Oct 29 22:13:56.084696 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:56.084658    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:13:56.084820 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:56.084737    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:13:56.084885 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:56.084846    4962 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 29 22:13:56.925315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:56.924772    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2" event={"ID":"71174a8c-e386-4670-942e-3bc750987080","Type":"ContainerStarted","Data":"0c1daee793a4f3895e641aab38ed7e988ab4dc2a0530cccf9d3942a518bd960e"}
Oct 29 22:13:56.925315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:56.924922    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2" event={"ID":"71174a8c-e386-4670-942e-3bc750987080","Type":"ContainerStarted","Data":"b8f0649fadcbfee3386c89a4f15bdf65ba2aee1f4532002785c048ddcb663838"}
Oct 29 22:13:57.753823 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:57.753789    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:13:57.755619 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:57.755592    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 29 22:13:57.755772 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:57.755651    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2" podUID="71174a8c-e386-4670-942e-3bc750987080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 29 22:13:58.757310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:58.756705    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 29 22:13:58.757310 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:58.756779    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2" podUID="71174a8c-e386-4670-942e-3bc750987080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 29 22:13:58.776311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:58.766852    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2" podStartSLOduration=3.766831894 podStartE2EDuration="3.766831894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:13:57.004684714 +0000 UTC m=+1658.077641704" watchObservedRunningTime="2026-10-29 22:13:58.766831894 +0000 UTC m=+1659.839788859"
Oct 29 22:13:58.776311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:58.772319    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"]
Oct 29 22:13:58.828312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:58.827352    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"]
Oct 29 22:13:59.039372 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:59.032826    4962 generic.go:356] "Generic (PLEG): container finished" podID="093f784b-0baf-4f14-984e-ec8003db30fb" containerID="97dd00635c0d257493e656a1cc325cac4e89fb9f39fead1fb6213c33bf7f7299" exitCode=0
Oct 29 22:13:59.048185 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:59.045353    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl" event={"ID":"093f784b-0baf-4f14-984e-ec8003db30fb","Type":"ContainerDied","Data":"97dd00635c0d257493e656a1cc325cac4e89fb9f39fead1fb6213c33bf7f7299"}
Oct 29 22:13:59.048185 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:59.045433    4962 scope.go:122] "RemoveContainer" containerID="c6930380d3da248a77f1ea09cc7b648d3d2f5b54d53a107c727312115a29c31e"
Oct 29 22:13:59.767320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:59.765345    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:13:59.823871 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:59.817343    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 29 22:13:59.823871 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:13:59.817399    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2" podUID="71174a8c-e386-4670-942e-3bc750987080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 29 22:14:00.446977 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.446946    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:14:00.550135 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.550097    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl" event={"ID":"093f784b-0baf-4f14-984e-ec8003db30fb","Type":"ContainerDied","Data":"f1eb1382333bc3871f4e52025f6dc44d13a97a7ad06dc96743e54187adad64ab"}
Oct 29 22:14:00.550464 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.550433    4962 scope.go:122] "RemoveContainer" containerID="97dd00635c0d257493e656a1cc325cac4e89fb9f39fead1fb6213c33bf7f7299"
Oct 29 22:14:00.550903 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.550877    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"
Oct 29 22:14:00.571018 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.570974    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/093f784b-0baf-4f14-984e-ec8003db30fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093f784b-0baf-4f14-984e-ec8003db30fb-istio-podinfo\") pod \"093f784b-0baf-4f14-984e-ec8003db30fb\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") "
Oct 29 22:14:00.571359 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.571339    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/093f784b-0baf-4f14-984e-ec8003db30fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093f784b-0baf-4f14-984e-ec8003db30fb-istiod-ca-cert\") pod \"093f784b-0baf-4f14-984e-ec8003db30fb\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") "
Oct 29 22:14:00.571642 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.571612    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-istio-envoy\") pod \"093f784b-0baf-4f14-984e-ec8003db30fb\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") "
Oct 29 22:14:00.571839 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.571809    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-credential-socket\") pod \"093f784b-0baf-4f14-984e-ec8003db30fb\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") "
Oct 29 22:14:00.592168 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.592137    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-workload-certs\") pod \"093f784b-0baf-4f14-984e-ec8003db30fb\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") "
Oct 29 22:14:00.592470 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.592435    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-istio-data\") pod \"093f784b-0baf-4f14-984e-ec8003db30fb\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") "
Oct 29 22:14:00.594142 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.594119    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093f784b-0baf-4f14-984e-ec8003db30fb-kube-api-access-kvhsz\" (UniqueName: \"kubernetes.io/projected/093f784b-0baf-4f14-984e-ec8003db30fb-kube-api-access-kvhsz\") pod \"093f784b-0baf-4f14-984e-ec8003db30fb\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") "
Oct 29 22:14:00.594370 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.594352    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-workload-socket\") pod \"093f784b-0baf-4f14-984e-ec8003db30fb\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") "
Oct 29 22:14:00.594658 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.594641    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093f784b-0baf-4f14-984e-ec8003db30fb-istio-token\" (UniqueName: \"kubernetes.io/projected/093f784b-0baf-4f14-984e-ec8003db30fb-istio-token\") pod \"093f784b-0baf-4f14-984e-ec8003db30fb\" (UID: \"093f784b-0baf-4f14-984e-ec8003db30fb\") "
Oct 29 22:14:00.601922 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.592951    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-istio-data" pod "093f784b-0baf-4f14-984e-ec8003db30fb" (UID: "093f784b-0baf-4f14-984e-ec8003db30fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:00.602052 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.593170    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-workload-certs" pod "093f784b-0baf-4f14-984e-ec8003db30fb" (UID: "093f784b-0baf-4f14-984e-ec8003db30fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:00.602367 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.593350    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-credential-socket" pod "093f784b-0baf-4f14-984e-ec8003db30fb" (UID: "093f784b-0baf-4f14-984e-ec8003db30fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:00.602497 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.600501    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093f784b-0baf-4f14-984e-ec8003db30fb-istiod-ca-cert" pod "093f784b-0baf-4f14-984e-ec8003db30fb" (UID: "093f784b-0baf-4f14-984e-ec8003db30fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:00.631621 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.631421    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-workload-socket" pod "093f784b-0baf-4f14-984e-ec8003db30fb" (UID: "093f784b-0baf-4f14-984e-ec8003db30fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:00.659783 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.659739    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093f784b-0baf-4f14-984e-ec8003db30fb-istio-token" pod "093f784b-0baf-4f14-984e-ec8003db30fb" (UID: "093f784b-0baf-4f14-984e-ec8003db30fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:00.694470 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.684447    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:14:00.694470 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.690404    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-istio-envoy" pod "093f784b-0baf-4f14-984e-ec8003db30fb" (UID: "093f784b-0baf-4f14-984e-ec8003db30fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:00.718549 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.718457    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093f784b-0baf-4f14-984e-ec8003db30fb-kube-api-access-kvhsz" pod "093f784b-0baf-4f14-984e-ec8003db30fb" (UID: "093f784b-0baf-4f14-984e-ec8003db30fb"). InnerVolumeSpecName "kube-api-access-kvhsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:00.765324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.742194    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvhsz\" (UniqueName: \"kubernetes.io/projected/093f784b-0baf-4f14-984e-ec8003db30fb-kube-api-access-kvhsz\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:00.765324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.742248    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:00.765324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.742267    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/093f784b-0baf-4f14-984e-ec8003db30fb-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:00.765324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.742281    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/093f784b-0baf-4f14-984e-ec8003db30fb-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:00.765324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.742315    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:00.765324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.742326    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:00.765324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.742337    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:00.765324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.742351    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/093f784b-0baf-4f14-984e-ec8003db30fb-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:00.765324 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.764477    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:00.766267 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.766237    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 29 22:14:00.766456 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.766429    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2" podUID="71174a8c-e386-4670-942e-3bc750987080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 29 22:14:00.789488 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.789380    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:00.982700 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.982602    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr" podUID="d0b72137-f812-4900-adce-9b88820a6473" containerName="payload-processing" containerID="cri-o://367cb5b6741d771dcc9eaeb7df1b875a741eed9f41c907dc2489aaf283ecbb18" gracePeriod=25
Oct 29 22:14:01.011314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:00.993445    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:14:01.011314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:01.002384    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq" podUID="fae18ff3-cbb6-4399-92e5-80145ffac570" containerName="payload-pre-processing" containerID="cri-o://bb368e9a8a6eeac04117dfe2a04806d067eb32548f0de23252260ea6539afda3" gracePeriod=25
Oct 29 22:14:01.155320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:01.152527    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/093f784b-0baf-4f14-984e-ec8003db30fb-istio-podinfo" pod "093f784b-0baf-4f14-984e-ec8003db30fb" (UID: "093f784b-0baf-4f14-984e-ec8003db30fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:01.239394 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:01.235118    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/093f784b-0baf-4f14-984e-ec8003db30fb-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:01.424507 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:01.424471    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"]
Oct 29 22:14:01.454312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:01.453819    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-4e8583-openshift-default-674465474f-zbzkl"]
Oct 29 22:14:01.715718 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:01.688596    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:01.715718 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:01.689022    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:01.715718 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:01.689780    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:14:01.817991 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:01.780608    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:01.834145 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:01.828454    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded"
Oct 29 22:14:01.834145 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:01.828982    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:14:01.834145 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:01.829499    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:14:01.888343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:01.866699    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 29 22:14:01.888343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:01.866771    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2" podUID="71174a8c-e386-4670-942e-3bc750987080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 29 22:14:01.888343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:01.868117    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": context deadline exceeded"
Oct 29 22:14:01.888343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:01.873147    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:14:02.035572 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:02.003440    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:02.035572 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:02.003889    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:14:02.689391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:02.689342    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:02.709652 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:02.708508    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:14:02.710258 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:02.710232    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 22:14:02.802960 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:02.802920    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 29 22:14:02.803250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:02.803216    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2" podUID="71174a8c-e386-4670-942e-3bc750987080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 29 22:14:02.829832 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:02.804029    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:03.179373 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:02.895850    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:03.179373 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:02.897061    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:14:03.179373 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:03.005594    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:03.755331 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:03.694792    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:03.815327 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:03.709494    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:03.815327 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:03.762409    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 29 22:14:03.815327 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:03.805221    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2" podUID="71174a8c-e386-4670-942e-3bc750987080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 29 22:14:03.815327 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:03.800210    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="093f784b-0baf-4f14-984e-ec8003db30fb" path="/var/lib/kubelet/pods/093f784b-0baf-4f14-984e-ec8003db30fb/volumes"
Oct 29 22:14:03.815327 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:03.806102    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:14:04.165449 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:04.165404    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:14:04.192735 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:04.172349    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:14:04.192735 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:04.172387    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": context deadline exceeded"
Oct 29 22:14:04.217383 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:04.203381    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:04.217585 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:04.217340    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:04.328509 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:04.321208    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:04.758308 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:04.758237    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 29 22:14:04.758537 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:04.758316    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2" podUID="71174a8c-e386-4670-942e-3bc750987080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 29 22:14:04.818311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:04.812552    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:04.818311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:04.812925    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:14:05.142367 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:05.141613    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq" podUID="fae18ff3-cbb6-4399-92e5-80145ffac570" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:14:05.179419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:05.179375    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:05.195319 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:05.180906    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:05.195915 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:05.187432    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:14:05.207307 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:05.204364    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:05.385312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:05.363673    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:14:05.385312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:05.363763    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf" podUID="75cf3e4f-0488-4210-ab72-17b7afdeda6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:05.757313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:05.753487    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 29 22:14:05.757623 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:05.757586    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2" podUID="71174a8c-e386-4670-942e-3bc750987080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 29 22:14:05.769328 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:05.767389    4962 patch_prober.go:29] interesting pod/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:14:05.769328 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:05.767443    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5" podUID="a437f4e2-dc10-41de-851a-ee488b08c6f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:05.825176 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:05.825133    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:14:05.825781 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:05.825749    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:14:06.085422 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:06.085325    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr" podUID="d0b72137-f812-4900-adce-9b88820a6473" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:14:06.162313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:06.158418    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:14:06.162313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:06.158487    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr" podUID="d91917d2-1f66-4ee4-a312-57398651739e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:06.190311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:06.189652    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:06.199371 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:06.197736    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:06.199371 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:06.198059    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": context deadline exceeded"
Oct 29 22:14:06.221312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:06.205312    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:06.742311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:06.737788    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:06.761580 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:06.761543    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 29 22:14:06.761901 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:06.761853    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2" podUID="71174a8c-e386-4670-942e-3bc750987080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 29 22:14:06.835502 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:06.835431    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:14:06.843353 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:06.835358    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:07.194455 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:07.193531    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:07.224323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:07.222489    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:07.224323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:07.222874    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:14:07.224323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:07.223143    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": context deadline exceeded"
Oct 29 22:14:07.755765 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:07.753699    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 29 22:14:07.755765 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:07.753762    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2" podUID="71174a8c-e386-4670-942e-3bc750987080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 29 22:14:07.855471 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:07.855412    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:14:07.868701 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:07.855734    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:08.195309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:08.194413    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:08.250312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:08.240019    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:14:08.250312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:08.240403    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:14:08.250312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:08.240674    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": context deadline exceeded"
Oct 29 22:14:08.875314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:08.870911    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:08.875314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:08.871485    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:09.004315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:09.003177    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 29 22:14:09.004315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:09.003249    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2" podUID="71174a8c-e386-4670-942e-3bc750987080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 29 22:14:09.197106 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:09.195971    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": context deadline exceeded"
Oct 29 22:14:09.254347 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:09.240162    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:09.255453 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:09.255414    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:09.256074 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:09.256037    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:14:09.760440 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:09.760402    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 29 22:14:09.760767 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:09.760708    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2" podUID="71174a8c-e386-4670-942e-3bc750987080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 29 22:14:09.878509 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:09.876442    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:09.878509 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:09.876853    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded"
Oct 29 22:14:10.142086 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:14:10.129075    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:14:10.197502 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:10.197436    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:10.247142 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:10.247099    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Oct 29 22:14:10.257057 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:10.257016    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:10.257623 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:10.257592    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": context deadline exceeded"
Oct 29 22:14:10.621311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:10.619126    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"]
Oct 29 22:14:10.621311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:10.619404    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866" podUID="ad28e0d7-1aaa-44b8-a284-d3383d08e5cb" containerName="maas-api" containerID="cri-o://6a820430aed12e8315bdc2f3c23dda981159dd1974cde4d43239e17c77083a1c" gracePeriod=30
Oct 29 22:14:10.805863 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:10.768216    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866" podUID="ad28e0d7-1aaa-44b8-a284-d3383d08e5cb" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.158:8443/health\": dial tcp 10.132.0.158:8443: connect: connection refused"
Oct 29 22:14:10.814677 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:10.806249    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:14:10.886744 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:10.878468    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:10.886744 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:10.879644    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Oct 29 22:14:11.229326 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:11.216685    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:11.298937 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:11.295616    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:11.298937 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:11.296999    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded"
Oct 29 22:14:11.298937 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:11.298469    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:11.553967 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:11.553858    4962 patch_prober.go:29] interesting pod/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:14:11.554257 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:11.554213    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5" podUID="9762d8d0-a759-42ac-bc1b-c948adfa6f9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:11.608371 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:11.608329    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 29 22:14:11.631802 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:11.608733    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2" podUID="71174a8c-e386-4670-942e-3bc750987080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 29 22:14:11.841187 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:11.782606    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 29 22:14:11.841187 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:11.782687    4962 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2" podUID="71174a8c-e386-4670-942e-3bc750987080" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 29 22:14:11.854368 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:11.853945    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded"
Oct 29 22:14:11.891070 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:11.890937    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:11.892820 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:11.892788    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:14:11.915577 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:11.914724    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:14:11.933378 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:11.933311    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 22:14:12.057414 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.057387    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:14:12.151366 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.148535    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:14:12.154716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.153568    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:14:12.155139 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.155116    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:14:12.337864 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.337838    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"
Oct 29 22:14:12.456875 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.456844    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-metrics-certs\") pod \"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb\" (UID: \"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb\") "
Oct 29 22:14:12.457053 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.456921    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-kube-api-access-k6x4g\" (UniqueName: \"kubernetes.io/projected/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-kube-api-access-k6x4g\") pod \"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb\" (UID: \"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb\") "
Oct 29 22:14:12.457102 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.457085    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-maas-api-tls\") pod \"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb\" (UID: \"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb\") "
Oct 29 22:14:12.457234 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.457211    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-openshift-service-ca\") pod \"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb\" (UID: \"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb\") "
Oct 29 22:14:12.457601 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.457578    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-openshift-service-ca" pod "ad28e0d7-1aaa-44b8-a284-d3383d08e5cb" (UID: "ad28e0d7-1aaa-44b8-a284-d3383d08e5cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:12.457927 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.457907    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:12.459200 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.459178    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-kube-api-access-k6x4g" pod "ad28e0d7-1aaa-44b8-a284-d3383d08e5cb" (UID: "ad28e0d7-1aaa-44b8-a284-d3383d08e5cb"). InnerVolumeSpecName "kube-api-access-k6x4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:12.459457 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.459438    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-maas-api-tls" pod "ad28e0d7-1aaa-44b8-a284-d3383d08e5cb" (UID: "ad28e0d7-1aaa-44b8-a284-d3383d08e5cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:12.459537 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.459458    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-metrics-certs" pod "ad28e0d7-1aaa-44b8-a284-d3383d08e5cb" (UID: "ad28e0d7-1aaa-44b8-a284-d3383d08e5cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:12.559881 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.559850    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:12.559881 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.559880    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6x4g\" (UniqueName: \"kubernetes.io/projected/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-kube-api-access-k6x4g\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:12.559881 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.559890    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:12.785844 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.785745    4962 generic.go:356] "Generic (PLEG): container finished" podID="ad28e0d7-1aaa-44b8-a284-d3383d08e5cb" containerID="6a820430aed12e8315bdc2f3c23dda981159dd1974cde4d43239e17c77083a1c" exitCode=0
Oct 29 22:14:12.785844 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.785798    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"
Oct 29 22:14:12.786037 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.785843    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866" event={"ID":"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb","Type":"ContainerDied","Data":"6a820430aed12e8315bdc2f3c23dda981159dd1974cde4d43239e17c77083a1c"}
Oct 29 22:14:12.786037 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.785919    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866" event={"ID":"ad28e0d7-1aaa-44b8-a284-d3383d08e5cb","Type":"ContainerDied","Data":"61cc5e3f1e618b95fe50f40a854b67eebc88a246e145d07015b263d314367047"}
Oct 29 22:14:12.786037 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.785957    4962 scope.go:122] "RemoveContainer" containerID="6a820430aed12e8315bdc2f3c23dda981159dd1974cde4d43239e17c77083a1c"
Oct 29 22:14:12.805713 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.805684    4962 scope.go:122] "RemoveContainer" containerID="6a820430aed12e8315bdc2f3c23dda981159dd1974cde4d43239e17c77083a1c"
Oct 29 22:14:12.806133 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:14:12.806032    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a820430aed12e8315bdc2f3c23dda981159dd1974cde4d43239e17c77083a1c\": container with ID starting with 6a820430aed12e8315bdc2f3c23dda981159dd1974cde4d43239e17c77083a1c not found: ID does not exist" containerID="6a820430aed12e8315bdc2f3c23dda981159dd1974cde4d43239e17c77083a1c"
Oct 29 22:14:12.806133 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.806089    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a820430aed12e8315bdc2f3c23dda981159dd1974cde4d43239e17c77083a1c"} err="failed to get container status \"6a820430aed12e8315bdc2f3c23dda981159dd1974cde4d43239e17c77083a1c\": rpc error: code = NotFound desc = could not find container \"6a820430aed12e8315bdc2f3c23dda981159dd1974cde4d43239e17c77083a1c\": container with ID starting with 6a820430aed12e8315bdc2f3c23dda981159dd1974cde4d43239e17c77083a1c not found: ID does not exist"
Oct 29 22:14:12.842808 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.842776    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"]
Oct 29 22:14:12.854556 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.854527    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b80d5b-655cf4958f-vh866"]
Oct 29 22:14:12.888341 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.888313    4962 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:14:12.889880 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:12.889853    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:14:13.187365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:13.177782    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad28e0d7-1aaa-44b8-a284-d3383d08e5cb" path="/var/lib/kubelet/pods/ad28e0d7-1aaa-44b8-a284-d3383d08e5cb/volumes"
Oct 29 22:14:15.043790 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:15.043749    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq" podUID="fae18ff3-cbb6-4399-92e5-80145ffac570" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:14:16.052664 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:16.052629    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr" podUID="d0b72137-f812-4900-adce-9b88820a6473" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:14:17.069712 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:17.069679    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd"]
Oct 29 22:14:17.070410 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:17.069911    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd" podUID="f84b8805-aa71-45c6-b053-526ac3787a08" containerName="maas-api" containerID="cri-o://f797c0ddf2aedb813cf70c2158ae0eb600dca7d6f3d43ede627228215898c3ab" gracePeriod=30
Oct 29 22:14:17.469327 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:17.464990    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp_e08900e4-1f62-471e-b4ca-c5ee2df27d1b/payload-processing/0.log"
Oct 29 22:14:17.703509 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:17.687991    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-d4fgq_3472b532-36f0-4a0d-9d04-27ec9a9f32e6/payload-processing/0.log"
Oct 29 22:14:17.997309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:17.993121    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-d4fgq_3472b532-36f0-4a0d-9d04-27ec9a9f32e6/payload-processing/0.log"
Oct 29 22:14:18.043247 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:18.042881    4962 generic.go:356] "Generic (PLEG): container finished" podID="f84b8805-aa71-45c6-b053-526ac3787a08" containerID="f797c0ddf2aedb813cf70c2158ae0eb600dca7d6f3d43ede627228215898c3ab" exitCode=0
Oct 29 22:14:18.043247 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:18.043080    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd" event={"ID":"f84b8805-aa71-45c6-b053-526ac3787a08","Type":"ContainerDied","Data":"f797c0ddf2aedb813cf70c2158ae0eb600dca7d6f3d43ede627228215898c3ab"}
Oct 29 22:14:18.054766 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:18.054723    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd"
Oct 29 22:14:18.229461 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:18.229428    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f84b8805-aa71-45c6-b053-526ac3787a08-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f84b8805-aa71-45c6-b053-526ac3787a08-metrics-certs\") pod \"f84b8805-aa71-45c6-b053-526ac3787a08\" (UID: \"f84b8805-aa71-45c6-b053-526ac3787a08\") "
Oct 29 22:14:18.230083 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:18.230058    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f84b8805-aa71-45c6-b053-526ac3787a08-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f84b8805-aa71-45c6-b053-526ac3787a08-openshift-service-ca\") pod \"f84b8805-aa71-45c6-b053-526ac3787a08\" (UID: \"f84b8805-aa71-45c6-b053-526ac3787a08\") "
Oct 29 22:14:18.230229 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:18.230214    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f84b8805-aa71-45c6-b053-526ac3787a08-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f84b8805-aa71-45c6-b053-526ac3787a08-maas-api-tls\") pod \"f84b8805-aa71-45c6-b053-526ac3787a08\" (UID: \"f84b8805-aa71-45c6-b053-526ac3787a08\") "
Oct 29 22:14:18.230420 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:18.230390    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f84b8805-aa71-45c6-b053-526ac3787a08-kube-api-access-l6bcv\" (UniqueName: \"kubernetes.io/projected/f84b8805-aa71-45c6-b053-526ac3787a08-kube-api-access-l6bcv\") pod \"f84b8805-aa71-45c6-b053-526ac3787a08\" (UID: \"f84b8805-aa71-45c6-b053-526ac3787a08\") "
Oct 29 22:14:18.233916 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:18.233892    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84b8805-aa71-45c6-b053-526ac3787a08-openshift-service-ca" pod "f84b8805-aa71-45c6-b053-526ac3787a08" (UID: "f84b8805-aa71-45c6-b053-526ac3787a08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:18.250349 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:18.247789    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f84b8805-aa71-45c6-b053-526ac3787a08-metrics-certs" pod "f84b8805-aa71-45c6-b053-526ac3787a08" (UID: "f84b8805-aa71-45c6-b053-526ac3787a08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:18.252199 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:18.252147    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84b8805-aa71-45c6-b053-526ac3787a08-kube-api-access-l6bcv" pod "f84b8805-aa71-45c6-b053-526ac3787a08" (UID: "f84b8805-aa71-45c6-b053-526ac3787a08"). InnerVolumeSpecName "kube-api-access-l6bcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:18.252756 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:18.252732    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f84b8805-aa71-45c6-b053-526ac3787a08-maas-api-tls" pod "f84b8805-aa71-45c6-b053-526ac3787a08" (UID: "f84b8805-aa71-45c6-b053-526ac3787a08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:18.333555 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:18.333519    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f84b8805-aa71-45c6-b053-526ac3787a08-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:18.333555 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:18.333552    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f84b8805-aa71-45c6-b053-526ac3787a08-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:18.333555 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:18.333562    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6bcv\" (UniqueName: \"kubernetes.io/projected/f84b8805-aa71-45c6-b053-526ac3787a08-kube-api-access-l6bcv\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:18.333795 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:18.333570    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f84b8805-aa71-45c6-b053-526ac3787a08-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:19.000826 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:19.000786    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"]
Oct 29 22:14:19.001073 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:19.001049    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk" podUID="baea6f61-3440-48ed-868f-7e2bbf77df97" containerName="istio-proxy" containerID="cri-o://f8c016cecf7de09ad241211316a0c3096fea936e87ec6be4422513fd7e300e6e" gracePeriod=30
Oct 29 22:14:19.055645 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:19.055595    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd"
Oct 29 22:14:19.059896 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:19.059870    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd" event={"ID":"f84b8805-aa71-45c6-b053-526ac3787a08","Type":"ContainerDied","Data":"e16edc60d54579110209728c34b709b73c714b459586227bf2e539ddec4d5f31"}
Oct 29 22:14:19.060035 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:19.059980    4962 scope.go:122] "RemoveContainer" containerID="f797c0ddf2aedb813cf70c2158ae0eb600dca7d6f3d43ede627228215898c3ab"
Oct 29 22:14:19.208312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:19.200362    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd"]
Oct 29 22:14:19.414908 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:19.414870    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b31f3f-8c46d45c6-kqvsd"]
Oct 29 22:14:21.060595 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:21.060568    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f84b8805-aa71-45c6-b053-526ac3787a08" path="/var/lib/kubelet/pods/f84b8805-aa71-45c6-b053-526ac3787a08/volumes"
Oct 29 22:14:21.312752 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:21.312730    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr"
Oct 29 22:14:21.346278 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:21.346256    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq"
Oct 29 22:14:21.376351 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:21.376325    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0b72137-f812-4900-adce-9b88820a6473-kube-api-access-2bmc6\" (UniqueName: \"kubernetes.io/projected/d0b72137-f812-4900-adce-9b88820a6473-kube-api-access-2bmc6\") pod \"d0b72137-f812-4900-adce-9b88820a6473\" (UID: \"d0b72137-f812-4900-adce-9b88820a6473\") "
Oct 29 22:14:21.376606 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:21.376589    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0b72137-f812-4900-adce-9b88820a6473-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0b72137-f812-4900-adce-9b88820a6473-plugins-config-volume\") pod \"d0b72137-f812-4900-adce-9b88820a6473\" (UID: \"d0b72137-f812-4900-adce-9b88820a6473\") "
Oct 29 22:14:21.376932 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:21.376909    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0b72137-f812-4900-adce-9b88820a6473-plugins-config-volume" pod "d0b72137-f812-4900-adce-9b88820a6473" (UID: "d0b72137-f812-4900-adce-9b88820a6473"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:21.377354 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:21.377333    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0b72137-f812-4900-adce-9b88820a6473-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:21.378518 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:21.378493    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0b72137-f812-4900-adce-9b88820a6473-kube-api-access-2bmc6" pod "d0b72137-f812-4900-adce-9b88820a6473" (UID: "d0b72137-f812-4900-adce-9b88820a6473"). InnerVolumeSpecName "kube-api-access-2bmc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:21.478879 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:21.478848    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae18ff3-cbb6-4399-92e5-80145ffac570-kube-api-access-68bm9\" (UniqueName: \"kubernetes.io/projected/fae18ff3-cbb6-4399-92e5-80145ffac570-kube-api-access-68bm9\") pod \"fae18ff3-cbb6-4399-92e5-80145ffac570\" (UID: \"fae18ff3-cbb6-4399-92e5-80145ffac570\") "
Oct 29 22:14:21.479036 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:21.478892    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fae18ff3-cbb6-4399-92e5-80145ffac570-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae18ff3-cbb6-4399-92e5-80145ffac570-plugins-config-volume\") pod \"fae18ff3-cbb6-4399-92e5-80145ffac570\" (UID: \"fae18ff3-cbb6-4399-92e5-80145ffac570\") "
Oct 29 22:14:21.479409 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:21.479380    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae18ff3-cbb6-4399-92e5-80145ffac570-plugins-config-volume" pod "fae18ff3-cbb6-4399-92e5-80145ffac570" (UID: "fae18ff3-cbb6-4399-92e5-80145ffac570"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:21.479608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:21.479592    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bmc6\" (UniqueName: \"kubernetes.io/projected/d0b72137-f812-4900-adce-9b88820a6473-kube-api-access-2bmc6\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:21.479654 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:21.479612    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fae18ff3-cbb6-4399-92e5-80145ffac570-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:21.481086 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:21.481059    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae18ff3-cbb6-4399-92e5-80145ffac570-kube-api-access-68bm9" pod "fae18ff3-cbb6-4399-92e5-80145ffac570" (UID: "fae18ff3-cbb6-4399-92e5-80145ffac570"). InnerVolumeSpecName "kube-api-access-68bm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:21.580908 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:21.580824    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68bm9\" (UniqueName: \"kubernetes.io/projected/fae18ff3-cbb6-4399-92e5-80145ffac570-kube-api-access-68bm9\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:22.076245 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:22.076213    4962 generic.go:356] "Generic (PLEG): container finished" podID="d0b72137-f812-4900-adce-9b88820a6473" containerID="367cb5b6741d771dcc9eaeb7df1b875a741eed9f41c907dc2489aaf283ecbb18" exitCode=0
Oct 29 22:14:22.076634 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:22.076254    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr"
Oct 29 22:14:22.076634 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:22.076327    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr" event={"ID":"d0b72137-f812-4900-adce-9b88820a6473","Type":"ContainerDied","Data":"367cb5b6741d771dcc9eaeb7df1b875a741eed9f41c907dc2489aaf283ecbb18"}
Oct 29 22:14:22.076634 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:22.076377    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr" event={"ID":"d0b72137-f812-4900-adce-9b88820a6473","Type":"ContainerDied","Data":"6336a14ee4e5b6c9c3a5395501d1941b6dccd0f3ab9625d9eea3845f42adb3ec"}
Oct 29 22:14:22.076634 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:22.076406    4962 scope.go:122] "RemoveContainer" containerID="367cb5b6741d771dcc9eaeb7df1b875a741eed9f41c907dc2489aaf283ecbb18"
Oct 29 22:14:22.077910 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:22.077883    4962 generic.go:356] "Generic (PLEG): container finished" podID="fae18ff3-cbb6-4399-92e5-80145ffac570" containerID="bb368e9a8a6eeac04117dfe2a04806d067eb32548f0de23252260ea6539afda3" exitCode=0
Oct 29 22:14:22.078038 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:22.077925    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq"
Oct 29 22:14:22.078038 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:22.078016    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq" event={"ID":"fae18ff3-cbb6-4399-92e5-80145ffac570","Type":"ContainerDied","Data":"bb368e9a8a6eeac04117dfe2a04806d067eb32548f0de23252260ea6539afda3"}
Oct 29 22:14:22.078152 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:22.078066    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq" event={"ID":"fae18ff3-cbb6-4399-92e5-80145ffac570","Type":"ContainerDied","Data":"eb3d96bc8743fbbeac4076571be8d8d5dd511ded3bd278e67d062fde45677785"}
Oct 29 22:14:22.088125 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:22.088084    4962 scope.go:122] "RemoveContainer" containerID="367cb5b6741d771dcc9eaeb7df1b875a741eed9f41c907dc2489aaf283ecbb18"
Oct 29 22:14:22.088371 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:14:22.088350    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"367cb5b6741d771dcc9eaeb7df1b875a741eed9f41c907dc2489aaf283ecbb18\": container with ID starting with 367cb5b6741d771dcc9eaeb7df1b875a741eed9f41c907dc2489aaf283ecbb18 not found: ID does not exist" containerID="367cb5b6741d771dcc9eaeb7df1b875a741eed9f41c907dc2489aaf283ecbb18"
Oct 29 22:14:22.088493 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:22.088377    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"367cb5b6741d771dcc9eaeb7df1b875a741eed9f41c907dc2489aaf283ecbb18"} err="failed to get container status \"367cb5b6741d771dcc9eaeb7df1b875a741eed9f41c907dc2489aaf283ecbb18\": rpc error: code = NotFound desc = could not find container \"367cb5b6741d771dcc9eaeb7df1b875a741eed9f41c907dc2489aaf283ecbb18\": container with ID starting with 367cb5b6741d771dcc9eaeb7df1b875a741eed9f41c907dc2489aaf283ecbb18 not found: ID does not exist"
Oct 29 22:14:22.088493 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:22.088392    4962 scope.go:122] "RemoveContainer" containerID="bb368e9a8a6eeac04117dfe2a04806d067eb32548f0de23252260ea6539afda3"
Oct 29 22:14:22.097315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:22.097297    4962 scope.go:122] "RemoveContainer" containerID="bb368e9a8a6eeac04117dfe2a04806d067eb32548f0de23252260ea6539afda3"
Oct 29 22:14:22.097581 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:14:22.097562    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb368e9a8a6eeac04117dfe2a04806d067eb32548f0de23252260ea6539afda3\": container with ID starting with bb368e9a8a6eeac04117dfe2a04806d067eb32548f0de23252260ea6539afda3 not found: ID does not exist" containerID="bb368e9a8a6eeac04117dfe2a04806d067eb32548f0de23252260ea6539afda3"
Oct 29 22:14:22.097648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:22.097588    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb368e9a8a6eeac04117dfe2a04806d067eb32548f0de23252260ea6539afda3"} err="failed to get container status \"bb368e9a8a6eeac04117dfe2a04806d067eb32548f0de23252260ea6539afda3\": rpc error: code = NotFound desc = could not find container \"bb368e9a8a6eeac04117dfe2a04806d067eb32548f0de23252260ea6539afda3\": container with ID starting with bb368e9a8a6eeac04117dfe2a04806d067eb32548f0de23252260ea6539afda3 not found: ID does not exist"
Oct 29 22:14:22.128460 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:22.128436    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq"]
Oct 29 22:14:22.139965 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:22.139946    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a940bc-87d7d7779-b4sfq"]
Oct 29 22:14:22.165971 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:22.165949    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr"]
Oct 29 22:14:22.171093 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:22.171070    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-a940bc-7db76d54cb-2s9jr"]
Oct 29 22:14:22.936844 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:22.936806    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:14:22.936993 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:22.936862    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk" podUID="baea6f61-3440-48ed-868f-7e2bbf77df97" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:14:23.059166 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:23.059132    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0b72137-f812-4900-adce-9b88820a6473" path="/var/lib/kubelet/pods/d0b72137-f812-4900-adce-9b88820a6473/volumes"
Oct 29 22:14:23.059666 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:23.059649    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fae18ff3-cbb6-4399-92e5-80145ffac570" path="/var/lib/kubelet/pods/fae18ff3-cbb6-4399-92e5-80145ffac570/volumes"
Oct 29 22:14:24.292791 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.292765    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:14:24.426477 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.426446    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-credential-socket\") pod \"baea6f61-3440-48ed-868f-7e2bbf77df97\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") "
Oct 29 22:14:24.426654 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.426498    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baea6f61-3440-48ed-868f-7e2bbf77df97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baea6f61-3440-48ed-868f-7e2bbf77df97-istiod-ca-cert\") pod \"baea6f61-3440-48ed-868f-7e2bbf77df97\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") "
Oct 29 22:14:24.426654 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.426540    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-workload-socket\") pod \"baea6f61-3440-48ed-868f-7e2bbf77df97\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") "
Oct 29 22:14:24.426654 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.426559    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-token\" (UniqueName: \"kubernetes.io/projected/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-token\") pod \"baea6f61-3440-48ed-868f-7e2bbf77df97\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") "
Oct 29 22:14:24.426654 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.426616    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baea6f61-3440-48ed-868f-7e2bbf77df97-kube-api-access-8cwff\" (UniqueName: \"kubernetes.io/projected/baea6f61-3440-48ed-868f-7e2bbf77df97-kube-api-access-8cwff\") pod \"baea6f61-3440-48ed-868f-7e2bbf77df97\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") "
Oct 29 22:14:24.426877 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.426672    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-podinfo\") pod \"baea6f61-3440-48ed-868f-7e2bbf77df97\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") "
Oct 29 22:14:24.426877 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.426733    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-workload-certs\") pod \"baea6f61-3440-48ed-868f-7e2bbf77df97\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") "
Oct 29 22:14:24.426877 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.426772    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-credential-socket" pod "baea6f61-3440-48ed-868f-7e2bbf77df97" (UID: "baea6f61-3440-48ed-868f-7e2bbf77df97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:24.427054 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.426883    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-data\") pod \"baea6f61-3440-48ed-868f-7e2bbf77df97\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") "
Oct 29 22:14:24.427054 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.426907    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-workload-socket" pod "baea6f61-3440-48ed-868f-7e2bbf77df97" (UID: "baea6f61-3440-48ed-868f-7e2bbf77df97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:24.427054 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.426930    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-envoy\") pod \"baea6f61-3440-48ed-868f-7e2bbf77df97\" (UID: \"baea6f61-3440-48ed-868f-7e2bbf77df97\") "
Oct 29 22:14:24.427194 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.427043    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-workload-certs" pod "baea6f61-3440-48ed-868f-7e2bbf77df97" (UID: "baea6f61-3440-48ed-868f-7e2bbf77df97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:24.427194 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.427080    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baea6f61-3440-48ed-868f-7e2bbf77df97-istiod-ca-cert" pod "baea6f61-3440-48ed-868f-7e2bbf77df97" (UID: "baea6f61-3440-48ed-868f-7e2bbf77df97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:24.427194 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.427098    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-data" pod "baea6f61-3440-48ed-868f-7e2bbf77df97" (UID: "baea6f61-3440-48ed-868f-7e2bbf77df97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:24.427817 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.427790    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:24.427945 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.427822    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:24.427945 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.427839    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:24.427945 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.427854    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baea6f61-3440-48ed-868f-7e2bbf77df97-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:24.427945 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.427867    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:24.429002 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.428980    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baea6f61-3440-48ed-868f-7e2bbf77df97-kube-api-access-8cwff" pod "baea6f61-3440-48ed-868f-7e2bbf77df97" (UID: "baea6f61-3440-48ed-868f-7e2bbf77df97"). InnerVolumeSpecName "kube-api-access-8cwff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:24.429127 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.429009    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-token" pod "baea6f61-3440-48ed-868f-7e2bbf77df97" (UID: "baea6f61-3440-48ed-868f-7e2bbf77df97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:24.429127 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.429030    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-envoy" pod "baea6f61-3440-48ed-868f-7e2bbf77df97" (UID: "baea6f61-3440-48ed-868f-7e2bbf77df97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:24.429127 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.429115    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-podinfo" pod "baea6f61-3440-48ed-868f-7e2bbf77df97" (UID: "baea6f61-3440-48ed-868f-7e2bbf77df97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:24.529741 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.529677    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cwff\" (UniqueName: \"kubernetes.io/projected/baea6f61-3440-48ed-868f-7e2bbf77df97-kube-api-access-8cwff\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:24.529741 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.529703    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:24.529741 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.529713    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:24.529741 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:24.529721    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baea6f61-3440-48ed-868f-7e2bbf77df97-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:25.186882 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:25.173968    4962 generic.go:356] "Generic (PLEG): container finished" podID="baea6f61-3440-48ed-868f-7e2bbf77df97" containerID="f8c016cecf7de09ad241211316a0c3096fea936e87ec6be4422513fd7e300e6e" exitCode=0
Oct 29 22:14:25.186882 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:25.174152    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk" event={"ID":"baea6f61-3440-48ed-868f-7e2bbf77df97","Type":"ContainerDied","Data":"f8c016cecf7de09ad241211316a0c3096fea936e87ec6be4422513fd7e300e6e"}
Oct 29 22:14:25.186882 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:25.174211    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk" event={"ID":"baea6f61-3440-48ed-868f-7e2bbf77df97","Type":"ContainerDied","Data":"17880cbc4daed9cad8d406f0300db6e231fce55fd0d78dfe9a4ebb029b0af954"}
Oct 29 22:14:25.186882 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:25.174246    4962 scope.go:122] "RemoveContainer" containerID="f8c016cecf7de09ad241211316a0c3096fea936e87ec6be4422513fd7e300e6e"
Oct 29 22:14:25.186882 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:25.174269    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"
Oct 29 22:14:25.203152 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:25.203122    4962 scope.go:122] "RemoveContainer" containerID="f8c016cecf7de09ad241211316a0c3096fea936e87ec6be4422513fd7e300e6e"
Oct 29 22:14:25.203562 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:14:25.203538    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c016cecf7de09ad241211316a0c3096fea936e87ec6be4422513fd7e300e6e\": container with ID starting with f8c016cecf7de09ad241211316a0c3096fea936e87ec6be4422513fd7e300e6e not found: ID does not exist" containerID="f8c016cecf7de09ad241211316a0c3096fea936e87ec6be4422513fd7e300e6e"
Oct 29 22:14:25.203646 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:25.203575    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c016cecf7de09ad241211316a0c3096fea936e87ec6be4422513fd7e300e6e"} err="failed to get container status \"f8c016cecf7de09ad241211316a0c3096fea936e87ec6be4422513fd7e300e6e\": rpc error: code = NotFound desc = could not find container \"f8c016cecf7de09ad241211316a0c3096fea936e87ec6be4422513fd7e300e6e\": container with ID starting with f8c016cecf7de09ad241211316a0c3096fea936e87ec6be4422513fd7e300e6e not found: ID does not exist"
Oct 29 22:14:25.246071 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:25.246045    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"]
Oct 29 22:14:25.259151 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:25.259115    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-b80d5b-openshift-default-5bc5ddc449-nd8fk"]
Oct 29 22:14:25.288952 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:25.288915    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"]
Oct 29 22:14:25.290017 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:25.289466    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf" podUID="75cf3e4f-0488-4210-ab72-17b7afdeda6a" containerName="istio-proxy" containerID="cri-o://446ddf4d2e4b52607bffde5137ed43ae0be0a4ec18d39c786d6b3cf653919e94" gracePeriod=30
Oct 29 22:14:25.298856 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:25.298827    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"]
Oct 29 22:14:25.299481 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:25.299346    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg" podUID="16540bea-446b-46d9-9cde-4c9c34785ce8" containerName="istio-proxy" containerID="cri-o://62b9ff8dbf2dcecae744e4db1bfaeecc0c29f79de97fed607153dbf915f5ec18" gracePeriod=30
Oct 29 22:14:25.305055 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:25.305031    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"]
Oct 29 22:14:25.305199 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:25.305180    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh" podUID="f06b758b-d09f-4882-a198-d88e91c7f390" containerName="istio-proxy" containerID="cri-o://e98eaf67c7a73837aedc222b40171dc7f8248fe0383cc321e7fb6b3861bf36f0" gracePeriod=30
Oct 29 22:14:25.311151 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:25.311128    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"]
Oct 29 22:14:25.311335 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:25.311280    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds" podUID="2a71a069-94f9-4736-8b9f-d43292554254" containerName="istio-proxy" containerID="cri-o://dcf87bbef5ce6b2f7eaaaa0ab0fdfa34fe4bf44ac593049d59c0c42307689d6d" gracePeriod=30
Oct 29 22:14:25.667742 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:25.667588    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:14:25.667742 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:25.667656    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds" podUID="2a71a069-94f9-4736-8b9f-d43292554254" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:14:26.092238 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:26.092162    4962 patch_prober.go:29] interesting pod/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:14:26.092238 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:26.092229    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg" podUID="16540bea-446b-46d9-9cde-4c9c34785ce8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:14:27.062490 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.062433    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main" containerID="cri-o://38678d15c466655bc92c1f48a898a191141840e68129c8fa0f6d70e03bbda5ad" gracePeriod=32
Oct 29 22:14:27.069165 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.069140    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baea6f61-3440-48ed-868f-7e2bbf77df97" path="/var/lib/kubelet/pods/baea6f61-3440-48ed-868f-7e2bbf77df97/volumes"
Oct 29 22:14:27.134615 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.134584    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main" containerID="cri-o://3591c3f23ae7dd91fc821238a9cc5895d79ca54231c801125ff9b0c5d7697e4c" gracePeriod=32
Oct 29 22:14:27.191339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.191276    4962 generic.go:356] "Generic (PLEG): container finished" podID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerID="38678d15c466655bc92c1f48a898a191141840e68129c8fa0f6d70e03bbda5ad" exitCode=0
Oct 29 22:14:27.191466 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.191311    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" event={"ID":"262b3d82-eee9-4a36-b7e3-244aa1641c07","Type":"ContainerDied","Data":"38678d15c466655bc92c1f48a898a191141840e68129c8fa0f6d70e03bbda5ad"}
Oct 29 22:14:27.450656 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.450630    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 22:14:27.453990 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.453969    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:14:27.511623 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.511593    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-model-cache\") pod \"262b3d82-eee9-4a36-b7e3-244aa1641c07\" (UID: \"262b3d82-eee9-4a36-b7e3-244aa1641c07\") "
Oct 29 22:14:27.511777 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.511687    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/262b3d82-eee9-4a36-b7e3-244aa1641c07-tls-certs\" (UniqueName: \"kubernetes.io/secret/262b3d82-eee9-4a36-b7e3-244aa1641c07-tls-certs\") pod \"262b3d82-eee9-4a36-b7e3-244aa1641c07\" (UID: \"262b3d82-eee9-4a36-b7e3-244aa1641c07\") "
Oct 29 22:14:27.511777 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.511764    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce61403b-7eab-4ba2-94e0-982fcea9e829-kube-api-access-f4989\" (UniqueName: \"kubernetes.io/projected/ce61403b-7eab-4ba2-94e0-982fcea9e829-kube-api-access-f4989\") pod \"ce61403b-7eab-4ba2-94e0-982fcea9e829\" (UID: \"ce61403b-7eab-4ba2-94e0-982fcea9e829\") "
Oct 29 22:14:27.511894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.511863    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-home\" (UniqueName: \"kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-home\") pod \"ce61403b-7eab-4ba2-94e0-982fcea9e829\" (UID: \"ce61403b-7eab-4ba2-94e0-982fcea9e829\") "
Oct 29 22:14:27.511957 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.511884    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-model-cache" pod "262b3d82-eee9-4a36-b7e3-244aa1641c07" (UID: "262b3d82-eee9-4a36-b7e3-244aa1641c07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:27.511957 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.511915    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-model-cache\") pod \"ce61403b-7eab-4ba2-94e0-982fcea9e829\" (UID: \"ce61403b-7eab-4ba2-94e0-982fcea9e829\") "
Oct 29 22:14:27.512078 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.511955    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/262b3d82-eee9-4a36-b7e3-244aa1641c07-kube-api-access-jdthz\" (UniqueName: \"kubernetes.io/projected/262b3d82-eee9-4a36-b7e3-244aa1641c07-kube-api-access-jdthz\") pod \"262b3d82-eee9-4a36-b7e3-244aa1641c07\" (UID: \"262b3d82-eee9-4a36-b7e3-244aa1641c07\") "
Oct 29 22:14:27.512078 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.512027    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-dshm\") pod \"ce61403b-7eab-4ba2-94e0-982fcea9e829\" (UID: \"ce61403b-7eab-4ba2-94e0-982fcea9e829\") "
Oct 29 22:14:27.512078 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.512060    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce61403b-7eab-4ba2-94e0-982fcea9e829-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce61403b-7eab-4ba2-94e0-982fcea9e829-tls-certs\") pod \"ce61403b-7eab-4ba2-94e0-982fcea9e829\" (UID: \"ce61403b-7eab-4ba2-94e0-982fcea9e829\") "
Oct 29 22:14:27.512249 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.512087    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-dshm\" (UniqueName: \"kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-dshm\") pod \"262b3d82-eee9-4a36-b7e3-244aa1641c07\" (UID: \"262b3d82-eee9-4a36-b7e3-244aa1641c07\") "
Oct 29 22:14:27.512249 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.512172    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-model-cache" pod "ce61403b-7eab-4ba2-94e0-982fcea9e829" (UID: "ce61403b-7eab-4ba2-94e0-982fcea9e829"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:27.512249 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.512236    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-tmp-dir\") pod \"ce61403b-7eab-4ba2-94e0-982fcea9e829\" (UID: \"ce61403b-7eab-4ba2-94e0-982fcea9e829\") "
Oct 29 22:14:27.512438 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.512321    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-home\" (UniqueName: \"kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-home\") pod \"262b3d82-eee9-4a36-b7e3-244aa1641c07\" (UID: \"262b3d82-eee9-4a36-b7e3-244aa1641c07\") "
Oct 29 22:14:27.512438 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.512370    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-tmp-dir\") pod \"262b3d82-eee9-4a36-b7e3-244aa1641c07\" (UID: \"262b3d82-eee9-4a36-b7e3-244aa1641c07\") "
Oct 29 22:14:27.512660 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.512638    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-home" pod "ce61403b-7eab-4ba2-94e0-982fcea9e829" (UID: "ce61403b-7eab-4ba2-94e0-982fcea9e829"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:27.512860 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.512778    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-home" pod "262b3d82-eee9-4a36-b7e3-244aa1641c07" (UID: "262b3d82-eee9-4a36-b7e3-244aa1641c07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:27.512939 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.512855    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-tmp-dir" pod "262b3d82-eee9-4a36-b7e3-244aa1641c07" (UID: "262b3d82-eee9-4a36-b7e3-244aa1641c07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:27.512939 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.512865    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-tmp-dir" pod "ce61403b-7eab-4ba2-94e0-982fcea9e829" (UID: "ce61403b-7eab-4ba2-94e0-982fcea9e829"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:27.513241 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.513220    4962 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-model-cache\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:27.513375 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.513246    4962 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-home\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:27.513375 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.513262    4962 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-model-cache\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:27.513375 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.513273    4962 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-tmp-dir\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:27.513375 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.513331    4962 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-home\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:27.513375 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.513344    4962 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-tmp-dir\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:27.516647 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.516625    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce61403b-7eab-4ba2-94e0-982fcea9e829-tls-certs" pod "ce61403b-7eab-4ba2-94e0-982fcea9e829" (UID: "ce61403b-7eab-4ba2-94e0-982fcea9e829"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:27.517962 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.517937    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-dshm" pod "262b3d82-eee9-4a36-b7e3-244aa1641c07" (UID: "262b3d82-eee9-4a36-b7e3-244aa1641c07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:27.518629 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.518590    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/262b3d82-eee9-4a36-b7e3-244aa1641c07-tls-certs" pod "262b3d82-eee9-4a36-b7e3-244aa1641c07" (UID: "262b3d82-eee9-4a36-b7e3-244aa1641c07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:27.518629 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.518613    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce61403b-7eab-4ba2-94e0-982fcea9e829-kube-api-access-f4989" pod "ce61403b-7eab-4ba2-94e0-982fcea9e829" (UID: "ce61403b-7eab-4ba2-94e0-982fcea9e829"). InnerVolumeSpecName "kube-api-access-f4989". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:27.518811 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.518642    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262b3d82-eee9-4a36-b7e3-244aa1641c07-kube-api-access-jdthz" pod "262b3d82-eee9-4a36-b7e3-244aa1641c07" (UID: "262b3d82-eee9-4a36-b7e3-244aa1641c07"). InnerVolumeSpecName "kube-api-access-jdthz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:27.519855 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.519832    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-dshm" pod "ce61403b-7eab-4ba2-94e0-982fcea9e829" (UID: "ce61403b-7eab-4ba2-94e0-982fcea9e829"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:27.614659 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.614582    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4989\" (UniqueName: \"kubernetes.io/projected/ce61403b-7eab-4ba2-94e0-982fcea9e829-kube-api-access-f4989\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:27.614659 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.614608    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdthz\" (UniqueName: \"kubernetes.io/projected/262b3d82-eee9-4a36-b7e3-244aa1641c07-kube-api-access-jdthz\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:27.614659 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.614618    4962 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce61403b-7eab-4ba2-94e0-982fcea9e829-dshm\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:27.614659 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.614627    4962 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce61403b-7eab-4ba2-94e0-982fcea9e829-tls-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:27.614659 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.614635    4962 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/262b3d82-eee9-4a36-b7e3-244aa1641c07-dshm\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:27.614659 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:27.614642    4962 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/262b3d82-eee9-4a36-b7e3-244aa1641c07-tls-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:28.207650 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.207617    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl"]
Oct 29 22:14:28.208465 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.208433    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl" podUID="49d31a1c-a9bf-413b-baf8-0ce2a310de72" containerName="maas-api" containerID="cri-o://e30b70dbb0efe8d96a328120d9ae91b763e48af18a99975cc3bae263d68e8dce" gracePeriod=30
Oct 29 22:14:28.210410 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.210358    4962 generic.go:356] "Generic (PLEG): container finished" podID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerID="3591c3f23ae7dd91fc821238a9cc5895d79ca54231c801125ff9b0c5d7697e4c" exitCode=0
Oct 29 22:14:28.210513 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.210497    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" event={"ID":"ce61403b-7eab-4ba2-94e0-982fcea9e829","Type":"ContainerDied","Data":"3591c3f23ae7dd91fc821238a9cc5895d79ca54231c801125ff9b0c5d7697e4c"}
Oct 29 22:14:28.210585 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.210524    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf" event={"ID":"ce61403b-7eab-4ba2-94e0-982fcea9e829","Type":"ContainerDied","Data":"74614c14e908aca3fe4bda53783672f3c4b058519dfed52eb9dbaa12d18340f7"}
Oct 29 22:14:28.210585 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.210557    4962 scope.go:122] "RemoveContainer" containerID="3591c3f23ae7dd91fc821238a9cc5895d79ca54231c801125ff9b0c5d7697e4c"
Oct 29 22:14:28.210716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.210666    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"
Oct 29 22:14:28.219463 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.218445    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b" event={"ID":"262b3d82-eee9-4a36-b7e3-244aa1641c07","Type":"ContainerDied","Data":"7ae4156820245c31d3e47405f4885e40f46fbee51d4bef4a24d043167df90b9b"}
Oct 29 22:14:28.219463 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.218624    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"
Oct 29 22:14:28.287922 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.287899    4962 scope.go:122] "RemoveContainer" containerID="3591c3f23ae7dd91fc821238a9cc5895d79ca54231c801125ff9b0c5d7697e4c"
Oct 29 22:14:28.288263 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:14:28.288241    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3591c3f23ae7dd91fc821238a9cc5895d79ca54231c801125ff9b0c5d7697e4c\": container with ID starting with 3591c3f23ae7dd91fc821238a9cc5895d79ca54231c801125ff9b0c5d7697e4c not found: ID does not exist" containerID="3591c3f23ae7dd91fc821238a9cc5895d79ca54231c801125ff9b0c5d7697e4c"
Oct 29 22:14:28.288351 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.288279    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3591c3f23ae7dd91fc821238a9cc5895d79ca54231c801125ff9b0c5d7697e4c"} err="failed to get container status \"3591c3f23ae7dd91fc821238a9cc5895d79ca54231c801125ff9b0c5d7697e4c\": rpc error: code = NotFound desc = could not find container \"3591c3f23ae7dd91fc821238a9cc5895d79ca54231c801125ff9b0c5d7697e4c\": container with ID starting with 3591c3f23ae7dd91fc821238a9cc5895d79ca54231c801125ff9b0c5d7697e4c not found: ID does not exist"
Oct 29 22:14:28.288351 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.288312    4962 scope.go:122] "RemoveContainer" containerID="38678d15c466655bc92c1f48a898a191141840e68129c8fa0f6d70e03bbda5ad"
Oct 29 22:14:28.324035 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.324009    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"]
Oct 29 22:14:28.328620 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.328589    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-743a5e/distinct-2-w5-743a5e-kserve-74bc67685d-zqfrf"]
Oct 29 22:14:28.363520 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.363490    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"]
Oct 29 22:14:28.372787 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.372685    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-743a5e/premium-w5-743a5e-kserve-d877dd66-lq47b"]
Oct 29 22:14:28.847577 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.847553    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl"
Oct 29 22:14:28.989998 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.989963    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49d31a1c-a9bf-413b-baf8-0ce2a310de72-metrics-certs\" (UniqueName: \"kubernetes.io/secret/49d31a1c-a9bf-413b-baf8-0ce2a310de72-metrics-certs\") pod \"49d31a1c-a9bf-413b-baf8-0ce2a310de72\" (UID: \"49d31a1c-a9bf-413b-baf8-0ce2a310de72\") "
Oct 29 22:14:28.989998 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.990002    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49d31a1c-a9bf-413b-baf8-0ce2a310de72-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d31a1c-a9bf-413b-baf8-0ce2a310de72-openshift-service-ca\") pod \"49d31a1c-a9bf-413b-baf8-0ce2a310de72\" (UID: \"49d31a1c-a9bf-413b-baf8-0ce2a310de72\") "
Oct 29 22:14:28.990256 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.990101    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49d31a1c-a9bf-413b-baf8-0ce2a310de72-kube-api-access-plq5r\" (UniqueName: \"kubernetes.io/projected/49d31a1c-a9bf-413b-baf8-0ce2a310de72-kube-api-access-plq5r\") pod \"49d31a1c-a9bf-413b-baf8-0ce2a310de72\" (UID: \"49d31a1c-a9bf-413b-baf8-0ce2a310de72\") "
Oct 29 22:14:28.990256 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.990150    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49d31a1c-a9bf-413b-baf8-0ce2a310de72-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49d31a1c-a9bf-413b-baf8-0ce2a310de72-maas-api-tls\") pod \"49d31a1c-a9bf-413b-baf8-0ce2a310de72\" (UID: \"49d31a1c-a9bf-413b-baf8-0ce2a310de72\") "
Oct 29 22:14:28.990500 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.990475    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49d31a1c-a9bf-413b-baf8-0ce2a310de72-openshift-service-ca" pod "49d31a1c-a9bf-413b-baf8-0ce2a310de72" (UID: "49d31a1c-a9bf-413b-baf8-0ce2a310de72"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:28.991107 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.991078    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d31a1c-a9bf-413b-baf8-0ce2a310de72-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:28.992553 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.992525    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49d31a1c-a9bf-413b-baf8-0ce2a310de72-maas-api-tls" pod "49d31a1c-a9bf-413b-baf8-0ce2a310de72" (UID: "49d31a1c-a9bf-413b-baf8-0ce2a310de72"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:28.992868 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.992836    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49d31a1c-a9bf-413b-baf8-0ce2a310de72-metrics-certs" pod "49d31a1c-a9bf-413b-baf8-0ce2a310de72" (UID: "49d31a1c-a9bf-413b-baf8-0ce2a310de72"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:28.993075 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:28.993051    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d31a1c-a9bf-413b-baf8-0ce2a310de72-kube-api-access-plq5r" pod "49d31a1c-a9bf-413b-baf8-0ce2a310de72" (UID: "49d31a1c-a9bf-413b-baf8-0ce2a310de72"). InnerVolumeSpecName "kube-api-access-plq5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:29.103325 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:29.101061    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plq5r\" (UniqueName: \"kubernetes.io/projected/49d31a1c-a9bf-413b-baf8-0ce2a310de72-kube-api-access-plq5r\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:29.103325 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:29.101104    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49d31a1c-a9bf-413b-baf8-0ce2a310de72-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:29.103325 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:29.101120    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49d31a1c-a9bf-413b-baf8-0ce2a310de72-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:29.183408 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:29.183372    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" path="/var/lib/kubelet/pods/262b3d82-eee9-4a36-b7e3-244aa1641c07/volumes"
Oct 29 22:14:29.184041 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:29.184015    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" path="/var/lib/kubelet/pods/ce61403b-7eab-4ba2-94e0-982fcea9e829/volumes"
Oct 29 22:14:29.230501 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:29.230462    4962 generic.go:356] "Generic (PLEG): container finished" podID="49d31a1c-a9bf-413b-baf8-0ce2a310de72" containerID="e30b70dbb0efe8d96a328120d9ae91b763e48af18a99975cc3bae263d68e8dce" exitCode=0
Oct 29 22:14:29.231098 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:29.230601    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl" event={"ID":"49d31a1c-a9bf-413b-baf8-0ce2a310de72","Type":"ContainerDied","Data":"e30b70dbb0efe8d96a328120d9ae91b763e48af18a99975cc3bae263d68e8dce"}
Oct 29 22:14:29.231098 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:29.230652    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl" event={"ID":"49d31a1c-a9bf-413b-baf8-0ce2a310de72","Type":"ContainerDied","Data":"c0a861d715160f8e237a645d8f77761469e61f58a2cae3f307c301a82a8027ff"}
Oct 29 22:14:29.231098 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:29.230680    4962 scope.go:122] "RemoveContainer" containerID="e30b70dbb0efe8d96a328120d9ae91b763e48af18a99975cc3bae263d68e8dce"
Oct 29 22:14:29.231098 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:29.230809    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl"
Oct 29 22:14:29.288363 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:29.283729    4962 scope.go:122] "RemoveContainer" containerID="e30b70dbb0efe8d96a328120d9ae91b763e48af18a99975cc3bae263d68e8dce"
Oct 29 22:14:29.288363 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:14:29.284077    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e30b70dbb0efe8d96a328120d9ae91b763e48af18a99975cc3bae263d68e8dce\": container with ID starting with e30b70dbb0efe8d96a328120d9ae91b763e48af18a99975cc3bae263d68e8dce not found: ID does not exist" containerID="e30b70dbb0efe8d96a328120d9ae91b763e48af18a99975cc3bae263d68e8dce"
Oct 29 22:14:29.288363 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:29.284106    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30b70dbb0efe8d96a328120d9ae91b763e48af18a99975cc3bae263d68e8dce"} err="failed to get container status \"e30b70dbb0efe8d96a328120d9ae91b763e48af18a99975cc3bae263d68e8dce\": rpc error: code = NotFound desc = could not find container \"e30b70dbb0efe8d96a328120d9ae91b763e48af18a99975cc3bae263d68e8dce\": container with ID starting with e30b70dbb0efe8d96a328120d9ae91b763e48af18a99975cc3bae263d68e8dce not found: ID does not exist"
Oct 29 22:14:29.330929 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:29.327417    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl"]
Oct 29 22:14:29.334943 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:29.332797    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-a940bc-99bdd9c74-pwnhl"]
Oct 29 22:14:30.627936 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.627609    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:14:30.630997 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.630967    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 22:14:30.721997 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.721973    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 22:14:30.725738 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.725711    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f06b758b-d09f-4882-a198-d88e91c7f390-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f06b758b-d09f-4882-a198-d88e91c7f390-istio-podinfo\") pod \"f06b758b-d09f-4882-a198-d88e91c7f390\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") "
Oct 29 22:14:30.725866 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.725781    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f06b758b-d09f-4882-a198-d88e91c7f390-istio-token\" (UniqueName: \"kubernetes.io/projected/f06b758b-d09f-4882-a198-d88e91c7f390-istio-token\") pod \"f06b758b-d09f-4882-a198-d88e91c7f390\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") "
Oct 29 22:14:30.725866 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.725817    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-istio-data\") pod \"2a71a069-94f9-4736-8b9f-d43292554254\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") "
Oct 29 22:14:30.725984 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.725875    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-workload-certs\") pod \"f06b758b-d09f-4882-a198-d88e91c7f390\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") "
Oct 29 22:14:30.725984 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.725920    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-istio-envoy\") pod \"f06b758b-d09f-4882-a198-d88e91c7f390\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") "
Oct 29 22:14:30.726179 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.726003    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-credential-socket\") pod \"f06b758b-d09f-4882-a198-d88e91c7f390\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") "
Oct 29 22:14:30.726179 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.726032    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a71a069-94f9-4736-8b9f-d43292554254-kube-api-access-k9mtz\" (UniqueName: \"kubernetes.io/projected/2a71a069-94f9-4736-8b9f-d43292554254-kube-api-access-k9mtz\") pod \"2a71a069-94f9-4736-8b9f-d43292554254\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") "
Oct 29 22:14:30.726179 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.726066    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a71a069-94f9-4736-8b9f-d43292554254-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a71a069-94f9-4736-8b9f-d43292554254-istiod-ca-cert\") pod \"2a71a069-94f9-4736-8b9f-d43292554254\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") "
Oct 29 22:14:30.726179 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.726141    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f06b758b-d09f-4882-a198-d88e91c7f390-kube-api-access-rhdvj\" (UniqueName: \"kubernetes.io/projected/f06b758b-d09f-4882-a198-d88e91c7f390-kube-api-access-rhdvj\") pod \"f06b758b-d09f-4882-a198-d88e91c7f390\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") "
Oct 29 22:14:30.726179 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.726170    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-workload-certs\") pod \"2a71a069-94f9-4736-8b9f-d43292554254\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") "
Oct 29 22:14:30.726493 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.726207    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-istio-data" pod "2a71a069-94f9-4736-8b9f-d43292554254" (UID: "2a71a069-94f9-4736-8b9f-d43292554254"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.726493 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.726266    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-workload-socket\") pod \"f06b758b-d09f-4882-a198-d88e91c7f390\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") "
Oct 29 22:14:30.726493 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.726334    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f06b758b-d09f-4882-a198-d88e91c7f390-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f06b758b-d09f-4882-a198-d88e91c7f390-istiod-ca-cert\") pod \"f06b758b-d09f-4882-a198-d88e91c7f390\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") "
Oct 29 22:14:30.726493 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.726423    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a71a069-94f9-4736-8b9f-d43292554254-istio-token\" (UniqueName: \"kubernetes.io/projected/2a71a069-94f9-4736-8b9f-d43292554254-istio-token\") pod \"2a71a069-94f9-4736-8b9f-d43292554254\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") "
Oct 29 22:14:30.726493 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.726433    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-workload-certs" pod "f06b758b-d09f-4882-a198-d88e91c7f390" (UID: "f06b758b-d09f-4882-a198-d88e91c7f390"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.726493 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.726493    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-credential-socket\") pod \"2a71a069-94f9-4736-8b9f-d43292554254\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") "
Oct 29 22:14:30.726824 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.726563    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a71a069-94f9-4736-8b9f-d43292554254-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a71a069-94f9-4736-8b9f-d43292554254-istio-podinfo\") pod \"2a71a069-94f9-4736-8b9f-d43292554254\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") "
Oct 29 22:14:30.726824 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.726592    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-istio-data\") pod \"f06b758b-d09f-4882-a198-d88e91c7f390\" (UID: \"f06b758b-d09f-4882-a198-d88e91c7f390\") "
Oct 29 22:14:30.726824 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.726621    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-istio-envoy\") pod \"2a71a069-94f9-4736-8b9f-d43292554254\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") "
Oct 29 22:14:30.726824 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.726632    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-workload-certs" pod "2a71a069-94f9-4736-8b9f-d43292554254" (UID: "2a71a069-94f9-4736-8b9f-d43292554254"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.726824 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.726748    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-workload-socket\") pod \"2a71a069-94f9-4736-8b9f-d43292554254\" (UID: \"2a71a069-94f9-4736-8b9f-d43292554254\") "
Oct 29 22:14:30.727111 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.726921    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-credential-socket" pod "f06b758b-d09f-4882-a198-d88e91c7f390" (UID: "f06b758b-d09f-4882-a198-d88e91c7f390"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.728614 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.728591    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.728724 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.728617    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.728724 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.728631    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.728724 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.728642    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.729010 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.728883    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-workload-socket" pod "2a71a069-94f9-4736-8b9f-d43292554254" (UID: "2a71a069-94f9-4736-8b9f-d43292554254"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.729653 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.729483    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-istio-data" pod "f06b758b-d09f-4882-a198-d88e91c7f390" (UID: "f06b758b-d09f-4882-a198-d88e91c7f390"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.729823 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.729694    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-istio-envoy" pod "2a71a069-94f9-4736-8b9f-d43292554254" (UID: "2a71a069-94f9-4736-8b9f-d43292554254"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.729979 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.729841    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-workload-socket" pod "f06b758b-d09f-4882-a198-d88e91c7f390" (UID: "f06b758b-d09f-4882-a198-d88e91c7f390"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.731090 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.730041    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-credential-socket" pod "2a71a069-94f9-4736-8b9f-d43292554254" (UID: "2a71a069-94f9-4736-8b9f-d43292554254"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.731090 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.730144    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a71a069-94f9-4736-8b9f-d43292554254-kube-api-access-k9mtz" pod "2a71a069-94f9-4736-8b9f-d43292554254" (UID: "2a71a069-94f9-4736-8b9f-d43292554254"). InnerVolumeSpecName "kube-api-access-k9mtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:30.731090 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.730173    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 22:14:30.731090 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.730306    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f06b758b-d09f-4882-a198-d88e91c7f390-istiod-ca-cert" pod "f06b758b-d09f-4882-a198-d88e91c7f390" (UID: "f06b758b-d09f-4882-a198-d88e91c7f390"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:30.731090 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.730669    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a71a069-94f9-4736-8b9f-d43292554254-istiod-ca-cert" pod "2a71a069-94f9-4736-8b9f-d43292554254" (UID: "2a71a069-94f9-4736-8b9f-d43292554254"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:30.731529 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.731369    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a71a069-94f9-4736-8b9f-d43292554254-istio-token" pod "2a71a069-94f9-4736-8b9f-d43292554254" (UID: "2a71a069-94f9-4736-8b9f-d43292554254"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:30.732183 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.732152    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a71a069-94f9-4736-8b9f-d43292554254-istio-podinfo" pod "2a71a069-94f9-4736-8b9f-d43292554254" (UID: "2a71a069-94f9-4736-8b9f-d43292554254"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:30.735339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.735234    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06b758b-d09f-4882-a198-d88e91c7f390-kube-api-access-rhdvj" pod "f06b758b-d09f-4882-a198-d88e91c7f390" (UID: "f06b758b-d09f-4882-a198-d88e91c7f390"). InnerVolumeSpecName "kube-api-access-rhdvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:30.735339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.735273    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06b758b-d09f-4882-a198-d88e91c7f390-istio-token" pod "f06b758b-d09f-4882-a198-d88e91c7f390" (UID: "f06b758b-d09f-4882-a198-d88e91c7f390"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:30.735518 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.735473    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f06b758b-d09f-4882-a198-d88e91c7f390-istio-podinfo" pod "f06b758b-d09f-4882-a198-d88e91c7f390" (UID: "f06b758b-d09f-4882-a198-d88e91c7f390"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:30.737180 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.737144    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-istio-envoy" pod "f06b758b-d09f-4882-a198-d88e91c7f390" (UID: "f06b758b-d09f-4882-a198-d88e91c7f390"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.829543 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.829451    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-podinfo\") pod \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") "
Oct 29 22:14:30.829543 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.829505    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-envoy\") pod \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") "
Oct 29 22:14:30.829826 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.829804    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-workload-socket\") pod \"16540bea-446b-46d9-9cde-4c9c34785ce8\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") "
Oct 29 22:14:30.829909 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.829850    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-data\") pod \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") "
Oct 29 22:14:30.829980 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.829925    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-data\") pod \"16540bea-446b-46d9-9cde-4c9c34785ce8\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") "
Oct 29 22:14:30.829980 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.829953    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-workload-certs\") pod \"16540bea-446b-46d9-9cde-4c9c34785ce8\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") "
Oct 29 22:14:30.830087 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.830008    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16540bea-446b-46d9-9cde-4c9c34785ce8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16540bea-446b-46d9-9cde-4c9c34785ce8-istiod-ca-cert\") pod \"16540bea-446b-46d9-9cde-4c9c34785ce8\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") "
Oct 29 22:14:30.830157 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.830092    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16540bea-446b-46d9-9cde-4c9c34785ce8-kube-api-access-6tv56\" (UniqueName: \"kubernetes.io/projected/16540bea-446b-46d9-9cde-4c9c34785ce8-kube-api-access-6tv56\") pod \"16540bea-446b-46d9-9cde-4c9c34785ce8\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") "
Oct 29 22:14:30.830157 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.830131    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-workload-certs\") pod \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") "
Oct 29 22:14:30.830278 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.830235    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-envoy\") pod \"16540bea-446b-46d9-9cde-4c9c34785ce8\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") "
Oct 29 22:14:30.830278 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.830248    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-data" pod "16540bea-446b-46d9-9cde-4c9c34785ce8" (UID: "16540bea-446b-46d9-9cde-4c9c34785ce8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.830410 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.830334    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-podinfo\") pod \"16540bea-446b-46d9-9cde-4c9c34785ce8\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") "
Oct 29 22:14:30.830410 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.830390    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-credential-socket\") pod \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") "
Oct 29 22:14:30.830524 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.830468    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istiod-ca-cert\") pod \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") "
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.830595    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-token\" (UniqueName: \"kubernetes.io/projected/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-token\") pod \"16540bea-446b-46d9-9cde-4c9c34785ce8\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") "
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.830632    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75cf3e4f-0488-4210-ab72-17b7afdeda6a-kube-api-access-ltzl8\" (UniqueName: \"kubernetes.io/projected/75cf3e4f-0488-4210-ab72-17b7afdeda6a-kube-api-access-ltzl8\") pod \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") "
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.830634    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-workload-socket" pod "16540bea-446b-46d9-9cde-4c9c34785ce8" (UID: "16540bea-446b-46d9-9cde-4c9c34785ce8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.830679    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-token\" (UniqueName: \"kubernetes.io/projected/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-token\") pod \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") "
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.830739    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-credential-socket\") pod \"16540bea-446b-46d9-9cde-4c9c34785ce8\" (UID: \"16540bea-446b-46d9-9cde-4c9c34785ce8\") "
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.830776    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-workload-socket\") pod \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\" (UID: \"75cf3e4f-0488-4210-ab72-17b7afdeda6a\") "
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.830972    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istiod-ca-cert" pod "75cf3e4f-0488-4210-ab72-17b7afdeda6a" (UID: "75cf3e4f-0488-4210-ab72-17b7afdeda6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.831202    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-credential-socket" pod "75cf3e4f-0488-4210-ab72-17b7afdeda6a" (UID: "75cf3e4f-0488-4210-ab72-17b7afdeda6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.831389    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-data" pod "75cf3e4f-0488-4210-ab72-17b7afdeda6a" (UID: "75cf3e4f-0488-4210-ab72-17b7afdeda6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.831568    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-workload-certs" pod "16540bea-446b-46d9-9cde-4c9c34785ce8" (UID: "16540bea-446b-46d9-9cde-4c9c34785ce8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.831582    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9mtz\" (UniqueName: \"kubernetes.io/projected/2a71a069-94f9-4736-8b9f-d43292554254-kube-api-access-k9mtz\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.831608    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a71a069-94f9-4736-8b9f-d43292554254-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.831622    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhdvj\" (UniqueName: \"kubernetes.io/projected/f06b758b-d09f-4882-a198-d88e91c7f390-kube-api-access-rhdvj\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.831636    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.831650    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.831663    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.831675    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.831688    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f06b758b-d09f-4882-a198-d88e91c7f390-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.831700    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a71a069-94f9-4736-8b9f-d43292554254-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.831712    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.831725    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a71a069-94f9-4736-8b9f-d43292554254-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.831737    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.831751    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.831762    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.831776    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a71a069-94f9-4736-8b9f-d43292554254-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.831789    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.831800    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f06b758b-d09f-4882-a198-d88e91c7f390-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.831811    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f06b758b-d09f-4882-a198-d88e91c7f390-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.831823    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f06b758b-d09f-4882-a198-d88e91c7f390-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.831854    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16540bea-446b-46d9-9cde-4c9c34785ce8-istiod-ca-cert" pod "16540bea-446b-46d9-9cde-4c9c34785ce8" (UID: "16540bea-446b-46d9-9cde-4c9c34785ce8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:30.832250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.832118    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-credential-socket" pod "16540bea-446b-46d9-9cde-4c9c34785ce8" (UID: "16540bea-446b-46d9-9cde-4c9c34785ce8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.833918 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.832307    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-workload-socket" pod "75cf3e4f-0488-4210-ab72-17b7afdeda6a" (UID: "75cf3e4f-0488-4210-ab72-17b7afdeda6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.833918 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.832998    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-workload-certs" pod "75cf3e4f-0488-4210-ab72-17b7afdeda6a" (UID: "75cf3e4f-0488-4210-ab72-17b7afdeda6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.833918 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.833151    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-podinfo" pod "16540bea-446b-46d9-9cde-4c9c34785ce8" (UID: "16540bea-446b-46d9-9cde-4c9c34785ce8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:30.834394 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.834367    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16540bea-446b-46d9-9cde-4c9c34785ce8-kube-api-access-6tv56" pod "16540bea-446b-46d9-9cde-4c9c34785ce8" (UID: "16540bea-446b-46d9-9cde-4c9c34785ce8"). InnerVolumeSpecName "kube-api-access-6tv56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:30.834524 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.834490    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-token" pod "75cf3e4f-0488-4210-ab72-17b7afdeda6a" (UID: "75cf3e4f-0488-4210-ab72-17b7afdeda6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:30.834613 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.834601    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-envoy" pod "75cf3e4f-0488-4210-ab72-17b7afdeda6a" (UID: "75cf3e4f-0488-4210-ab72-17b7afdeda6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.834893 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.834873    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-podinfo" pod "75cf3e4f-0488-4210-ab72-17b7afdeda6a" (UID: "75cf3e4f-0488-4210-ab72-17b7afdeda6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:30.834973 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.834890    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-envoy" pod "16540bea-446b-46d9-9cde-4c9c34785ce8" (UID: "16540bea-446b-46d9-9cde-4c9c34785ce8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:30.835597 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.835579    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75cf3e4f-0488-4210-ab72-17b7afdeda6a-kube-api-access-ltzl8" pod "75cf3e4f-0488-4210-ab72-17b7afdeda6a" (UID: "75cf3e4f-0488-4210-ab72-17b7afdeda6a"). InnerVolumeSpecName "kube-api-access-ltzl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:30.836163 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.836148    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-token" pod "16540bea-446b-46d9-9cde-4c9c34785ce8" (UID: "16540bea-446b-46d9-9cde-4c9c34785ce8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:30.932830 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.932798    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.932830 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.932829    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltzl8\" (UniqueName: \"kubernetes.io/projected/75cf3e4f-0488-4210-ab72-17b7afdeda6a-kube-api-access-ltzl8\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.932830 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.932839    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.933061 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.932848    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.933061 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.932856    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.933061 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.932864    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.933061 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.932875    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.933061 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.932883    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.933061 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.932890    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16540bea-446b-46d9-9cde-4c9c34785ce8-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.933061 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.932898    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tv56\" (UniqueName: \"kubernetes.io/projected/16540bea-446b-46d9-9cde-4c9c34785ce8-kube-api-access-6tv56\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.933061 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.932907    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75cf3e4f-0488-4210-ab72-17b7afdeda6a-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.933061 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.932916    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:30.933061 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:30.932924    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16540bea-446b-46d9-9cde-4c9c34785ce8-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:31.059594 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.059564    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49d31a1c-a9bf-413b-baf8-0ce2a310de72" path="/var/lib/kubelet/pods/49d31a1c-a9bf-413b-baf8-0ce2a310de72/volumes"
Oct 29 22:14:31.301145 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.301115    4962 generic.go:356] "Generic (PLEG): container finished" podID="f06b758b-d09f-4882-a198-d88e91c7f390" containerID="e98eaf67c7a73837aedc222b40171dc7f8248fe0383cc321e7fb6b3861bf36f0" exitCode=0
Oct 29 22:14:31.301365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.301173    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"
Oct 29 22:14:31.301365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.301196    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh" event={"ID":"f06b758b-d09f-4882-a198-d88e91c7f390","Type":"ContainerDied","Data":"e98eaf67c7a73837aedc222b40171dc7f8248fe0383cc321e7fb6b3861bf36f0"}
Oct 29 22:14:31.301365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.301281    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh" event={"ID":"f06b758b-d09f-4882-a198-d88e91c7f390","Type":"ContainerDied","Data":"a6dbed014094c123f4b7fe0106d58deb9157fd004f814d465cf439fc4765ffc2"}
Oct 29 22:14:31.301365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.301347    4962 scope.go:122] "RemoveContainer" containerID="e98eaf67c7a73837aedc222b40171dc7f8248fe0383cc321e7fb6b3861bf36f0"
Oct 29 22:14:31.303003 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.302978    4962 generic.go:356] "Generic (PLEG): container finished" podID="16540bea-446b-46d9-9cde-4c9c34785ce8" containerID="62b9ff8dbf2dcecae744e4db1bfaeecc0c29f79de97fed607153dbf915f5ec18" exitCode=0
Oct 29 22:14:31.303129 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.303031    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"
Oct 29 22:14:31.303129 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.303047    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg" event={"ID":"16540bea-446b-46d9-9cde-4c9c34785ce8","Type":"ContainerDied","Data":"62b9ff8dbf2dcecae744e4db1bfaeecc0c29f79de97fed607153dbf915f5ec18"}
Oct 29 22:14:31.303129 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.303095    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg" event={"ID":"16540bea-446b-46d9-9cde-4c9c34785ce8","Type":"ContainerDied","Data":"4f2526809fa47c0d68df65f7bfcd32b72b0039c3f2c52a539103ce864273c176"}
Oct 29 22:14:31.304854 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.304829    4962 generic.go:356] "Generic (PLEG): container finished" podID="2a71a069-94f9-4736-8b9f-d43292554254" containerID="dcf87bbef5ce6b2f7eaaaa0ab0fdfa34fe4bf44ac593049d59c0c42307689d6d" exitCode=0
Oct 29 22:14:31.304965 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.304879    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"
Oct 29 22:14:31.305036 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.304979    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds" event={"ID":"2a71a069-94f9-4736-8b9f-d43292554254","Type":"ContainerDied","Data":"dcf87bbef5ce6b2f7eaaaa0ab0fdfa34fe4bf44ac593049d59c0c42307689d6d"}
Oct 29 22:14:31.305099 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.305041    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds" event={"ID":"2a71a069-94f9-4736-8b9f-d43292554254","Type":"ContainerDied","Data":"9173e0d0267144bf05845444c74b2cd68fa38d6395c6d2c9f94fec8fb8502442"}
Oct 29 22:14:31.307066 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.307011    4962 generic.go:356] "Generic (PLEG): container finished" podID="75cf3e4f-0488-4210-ab72-17b7afdeda6a" containerID="446ddf4d2e4b52607bffde5137ed43ae0be0a4ec18d39c786d6b3cf653919e94" exitCode=0
Oct 29 22:14:31.307066 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.307039    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf" event={"ID":"75cf3e4f-0488-4210-ab72-17b7afdeda6a","Type":"ContainerDied","Data":"446ddf4d2e4b52607bffde5137ed43ae0be0a4ec18d39c786d6b3cf653919e94"}
Oct 29 22:14:31.307190 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.307084    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"
Oct 29 22:14:31.307190 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.307085    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf" event={"ID":"75cf3e4f-0488-4210-ab72-17b7afdeda6a","Type":"ContainerDied","Data":"27b698f1abec76abc65b36051054fc42f0e41c7e2700a6681bb798bb0fa5f036"}
Oct 29 22:14:31.313591 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.313575    4962 scope.go:122] "RemoveContainer" containerID="e98eaf67c7a73837aedc222b40171dc7f8248fe0383cc321e7fb6b3861bf36f0"
Oct 29 22:14:31.313859 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:14:31.313840    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e98eaf67c7a73837aedc222b40171dc7f8248fe0383cc321e7fb6b3861bf36f0\": container with ID starting with e98eaf67c7a73837aedc222b40171dc7f8248fe0383cc321e7fb6b3861bf36f0 not found: ID does not exist" containerID="e98eaf67c7a73837aedc222b40171dc7f8248fe0383cc321e7fb6b3861bf36f0"
Oct 29 22:14:31.313905 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.313867    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e98eaf67c7a73837aedc222b40171dc7f8248fe0383cc321e7fb6b3861bf36f0"} err="failed to get container status \"e98eaf67c7a73837aedc222b40171dc7f8248fe0383cc321e7fb6b3861bf36f0\": rpc error: code = NotFound desc = could not find container \"e98eaf67c7a73837aedc222b40171dc7f8248fe0383cc321e7fb6b3861bf36f0\": container with ID starting with e98eaf67c7a73837aedc222b40171dc7f8248fe0383cc321e7fb6b3861bf36f0 not found: ID does not exist"
Oct 29 22:14:31.313905 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.313883    4962 scope.go:122] "RemoveContainer" containerID="62b9ff8dbf2dcecae744e4db1bfaeecc0c29f79de97fed607153dbf915f5ec18"
Oct 29 22:14:31.349347 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.349327    4962 scope.go:122] "RemoveContainer" containerID="62b9ff8dbf2dcecae744e4db1bfaeecc0c29f79de97fed607153dbf915f5ec18"
Oct 29 22:14:31.349697 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:14:31.349670    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b9ff8dbf2dcecae744e4db1bfaeecc0c29f79de97fed607153dbf915f5ec18\": container with ID starting with 62b9ff8dbf2dcecae744e4db1bfaeecc0c29f79de97fed607153dbf915f5ec18 not found: ID does not exist" containerID="62b9ff8dbf2dcecae744e4db1bfaeecc0c29f79de97fed607153dbf915f5ec18"
Oct 29 22:14:31.349802 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.349706    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b9ff8dbf2dcecae744e4db1bfaeecc0c29f79de97fed607153dbf915f5ec18"} err="failed to get container status \"62b9ff8dbf2dcecae744e4db1bfaeecc0c29f79de97fed607153dbf915f5ec18\": rpc error: code = NotFound desc = could not find container \"62b9ff8dbf2dcecae744e4db1bfaeecc0c29f79de97fed607153dbf915f5ec18\": container with ID starting with 62b9ff8dbf2dcecae744e4db1bfaeecc0c29f79de97fed607153dbf915f5ec18 not found: ID does not exist"
Oct 29 22:14:31.349802 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.349729    4962 scope.go:122] "RemoveContainer" containerID="dcf87bbef5ce6b2f7eaaaa0ab0fdfa34fe4bf44ac593049d59c0c42307689d6d"
Oct 29 22:14:31.352945 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.352924    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"]
Oct 29 22:14:31.359434 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.359403    4962 scope.go:122] "RemoveContainer" containerID="dcf87bbef5ce6b2f7eaaaa0ab0fdfa34fe4bf44ac593049d59c0c42307689d6d"
Oct 29 22:14:31.359723 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:14:31.359703    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcf87bbef5ce6b2f7eaaaa0ab0fdfa34fe4bf44ac593049d59c0c42307689d6d\": container with ID starting with dcf87bbef5ce6b2f7eaaaa0ab0fdfa34fe4bf44ac593049d59c0c42307689d6d not found: ID does not exist" containerID="dcf87bbef5ce6b2f7eaaaa0ab0fdfa34fe4bf44ac593049d59c0c42307689d6d"
Oct 29 22:14:31.359810 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.359730    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcf87bbef5ce6b2f7eaaaa0ab0fdfa34fe4bf44ac593049d59c0c42307689d6d"} err="failed to get container status \"dcf87bbef5ce6b2f7eaaaa0ab0fdfa34fe4bf44ac593049d59c0c42307689d6d\": rpc error: code = NotFound desc = could not find container \"dcf87bbef5ce6b2f7eaaaa0ab0fdfa34fe4bf44ac593049d59c0c42307689d6d\": container with ID starting with dcf87bbef5ce6b2f7eaaaa0ab0fdfa34fe4bf44ac593049d59c0c42307689d6d not found: ID does not exist"
Oct 29 22:14:31.359810 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.359753    4962 scope.go:122] "RemoveContainer" containerID="446ddf4d2e4b52607bffde5137ed43ae0be0a4ec18d39c786d6b3cf653919e94"
Oct 29 22:14:31.364594 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.364574    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-wpxds"]
Oct 29 22:14:31.369248 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.369231    4962 scope.go:122] "RemoveContainer" containerID="09a8d0a86523179a06b9f2864e8a2ed76f2894553b2d17cad788c3c674d66ab6"
Oct 29 22:14:31.378157 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.378139    4962 scope.go:122] "RemoveContainer" containerID="446ddf4d2e4b52607bffde5137ed43ae0be0a4ec18d39c786d6b3cf653919e94"
Oct 29 22:14:31.378453 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:14:31.378432    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"446ddf4d2e4b52607bffde5137ed43ae0be0a4ec18d39c786d6b3cf653919e94\": container with ID starting with 446ddf4d2e4b52607bffde5137ed43ae0be0a4ec18d39c786d6b3cf653919e94 not found: ID does not exist" containerID="446ddf4d2e4b52607bffde5137ed43ae0be0a4ec18d39c786d6b3cf653919e94"
Oct 29 22:14:31.378526 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.378461    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"446ddf4d2e4b52607bffde5137ed43ae0be0a4ec18d39c786d6b3cf653919e94"} err="failed to get container status \"446ddf4d2e4b52607bffde5137ed43ae0be0a4ec18d39c786d6b3cf653919e94\": rpc error: code = NotFound desc = could not find container \"446ddf4d2e4b52607bffde5137ed43ae0be0a4ec18d39c786d6b3cf653919e94\": container with ID starting with 446ddf4d2e4b52607bffde5137ed43ae0be0a4ec18d39c786d6b3cf653919e94 not found: ID does not exist"
Oct 29 22:14:31.378526 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.378476    4962 scope.go:122] "RemoveContainer" containerID="09a8d0a86523179a06b9f2864e8a2ed76f2894553b2d17cad788c3c674d66ab6"
Oct 29 22:14:31.378747 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:14:31.378732    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09a8d0a86523179a06b9f2864e8a2ed76f2894553b2d17cad788c3c674d66ab6\": container with ID starting with 09a8d0a86523179a06b9f2864e8a2ed76f2894553b2d17cad788c3c674d66ab6 not found: ID does not exist" containerID="09a8d0a86523179a06b9f2864e8a2ed76f2894553b2d17cad788c3c674d66ab6"
Oct 29 22:14:31.378789 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.378750    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09a8d0a86523179a06b9f2864e8a2ed76f2894553b2d17cad788c3c674d66ab6"} err="failed to get container status \"09a8d0a86523179a06b9f2864e8a2ed76f2894553b2d17cad788c3c674d66ab6\": rpc error: code = NotFound desc = could not find container \"09a8d0a86523179a06b9f2864e8a2ed76f2894553b2d17cad788c3c674d66ab6\": container with ID starting with 09a8d0a86523179a06b9f2864e8a2ed76f2894553b2d17cad788c3c674d66ab6 not found: ID does not exist"
Oct 29 22:14:31.388940 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.388916    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"]
Oct 29 22:14:31.392459 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.392440    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-pgxgf"]
Oct 29 22:14:31.425335 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.425242    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"]
Oct 29 22:14:31.428720 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.428697    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-bkwdh"]
Oct 29 22:14:31.460743 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.460722    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"]
Oct 29 22:14:31.464919 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:31.464890    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b31f3f-openshift-default-55bc8ff4ff-27krg"]
Oct 29 22:14:33.059175 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:33.059143    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16540bea-446b-46d9-9cde-4c9c34785ce8" path="/var/lib/kubelet/pods/16540bea-446b-46d9-9cde-4c9c34785ce8/volumes"
Oct 29 22:14:33.059707 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:33.059685    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a71a069-94f9-4736-8b9f-d43292554254" path="/var/lib/kubelet/pods/2a71a069-94f9-4736-8b9f-d43292554254/volumes"
Oct 29 22:14:33.060155 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:33.060142    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75cf3e4f-0488-4210-ab72-17b7afdeda6a" path="/var/lib/kubelet/pods/75cf3e4f-0488-4210-ab72-17b7afdeda6a/volumes"
Oct 29 22:14:33.060934 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:33.060916    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f06b758b-d09f-4882-a198-d88e91c7f390" path="/var/lib/kubelet/pods/f06b758b-d09f-4882-a198-d88e91c7f390/volumes"
Oct 29 22:14:33.346362 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:33.343836    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc"]
Oct 29 22:14:33.346362 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:33.344107    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc" podUID="f18a68f6-5a07-41c2-acbe-58a2c98ebdd7" containerName="maas-api" containerID="cri-o://5214eb98fa1e5c7c76d8b3310b6504e1cdbab340877b081291304e44b2a36980" gracePeriod=30
Oct 29 22:14:34.118309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:34.115497    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc"
Oct 29 22:14:34.217739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:34.217056    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-openshift-service-ca\") pod \"f18a68f6-5a07-41c2-acbe-58a2c98ebdd7\" (UID: \"f18a68f6-5a07-41c2-acbe-58a2c98ebdd7\") "
Oct 29 22:14:34.217739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:34.217256    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-metrics-certs\") pod \"f18a68f6-5a07-41c2-acbe-58a2c98ebdd7\" (UID: \"f18a68f6-5a07-41c2-acbe-58a2c98ebdd7\") "
Oct 29 22:14:34.217739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:34.217345    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-maas-api-tls\") pod \"f18a68f6-5a07-41c2-acbe-58a2c98ebdd7\" (UID: \"f18a68f6-5a07-41c2-acbe-58a2c98ebdd7\") "
Oct 29 22:14:34.217739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:34.217449    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-kube-api-access-6dtz2\" (UniqueName: \"kubernetes.io/projected/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-kube-api-access-6dtz2\") pod \"f18a68f6-5a07-41c2-acbe-58a2c98ebdd7\" (UID: \"f18a68f6-5a07-41c2-acbe-58a2c98ebdd7\") "
Oct 29 22:14:34.227527 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:34.227494    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-openshift-service-ca" pod "f18a68f6-5a07-41c2-acbe-58a2c98ebdd7" (UID: "f18a68f6-5a07-41c2-acbe-58a2c98ebdd7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:34.249900 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:34.249794    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-metrics-certs" pod "f18a68f6-5a07-41c2-acbe-58a2c98ebdd7" (UID: "f18a68f6-5a07-41c2-acbe-58a2c98ebdd7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:34.250501 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:34.250472    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-kube-api-access-6dtz2" pod "f18a68f6-5a07-41c2-acbe-58a2c98ebdd7" (UID: "f18a68f6-5a07-41c2-acbe-58a2c98ebdd7"). InnerVolumeSpecName "kube-api-access-6dtz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:34.253124 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:34.253079    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-maas-api-tls" pod "f18a68f6-5a07-41c2-acbe-58a2c98ebdd7" (UID: "f18a68f6-5a07-41c2-acbe-58a2c98ebdd7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:34.325839 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:34.325802    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:34.325839 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:34.325835    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:34.325839 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:34.325845    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dtz2\" (UniqueName: \"kubernetes.io/projected/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-kube-api-access-6dtz2\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:34.326089 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:34.325855    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:34.561438 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:34.561353    4962 generic.go:356] "Generic (PLEG): container finished" podID="f18a68f6-5a07-41c2-acbe-58a2c98ebdd7" containerID="5214eb98fa1e5c7c76d8b3310b6504e1cdbab340877b081291304e44b2a36980" exitCode=0
Oct 29 22:14:34.561438 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:34.561407    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc"
Oct 29 22:14:34.561605 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:34.561439    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc" event={"ID":"f18a68f6-5a07-41c2-acbe-58a2c98ebdd7","Type":"ContainerDied","Data":"5214eb98fa1e5c7c76d8b3310b6504e1cdbab340877b081291304e44b2a36980"}
Oct 29 22:14:34.561605 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:34.561495    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc" event={"ID":"f18a68f6-5a07-41c2-acbe-58a2c98ebdd7","Type":"ContainerDied","Data":"a9a03b9165c8e69ea5c465149b70747fc397ddf70e85e62ff0ebc4cf089ee9f6"}
Oct 29 22:14:34.561605 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:34.561517    4962 scope.go:122] "RemoveContainer" containerID="5214eb98fa1e5c7c76d8b3310b6504e1cdbab340877b081291304e44b2a36980"
Oct 29 22:14:34.571990 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:34.571974    4962 scope.go:122] "RemoveContainer" containerID="5214eb98fa1e5c7c76d8b3310b6504e1cdbab340877b081291304e44b2a36980"
Oct 29 22:14:34.572259 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:14:34.572238    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5214eb98fa1e5c7c76d8b3310b6504e1cdbab340877b081291304e44b2a36980\": container with ID starting with 5214eb98fa1e5c7c76d8b3310b6504e1cdbab340877b081291304e44b2a36980 not found: ID does not exist" containerID="5214eb98fa1e5c7c76d8b3310b6504e1cdbab340877b081291304e44b2a36980"
Oct 29 22:14:34.572362 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:34.572266    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5214eb98fa1e5c7c76d8b3310b6504e1cdbab340877b081291304e44b2a36980"} err="failed to get container status \"5214eb98fa1e5c7c76d8b3310b6504e1cdbab340877b081291304e44b2a36980\": rpc error: code = NotFound desc = could not find container \"5214eb98fa1e5c7c76d8b3310b6504e1cdbab340877b081291304e44b2a36980\": container with ID starting with 5214eb98fa1e5c7c76d8b3310b6504e1cdbab340877b081291304e44b2a36980 not found: ID does not exist"
Oct 29 22:14:34.611912 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:34.611892    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc"]
Oct 29 22:14:34.625126 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:34.625097    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-75db5b-6c95f654b-z6vsc"]
Oct 29 22:14:35.066297 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:35.066258    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f18a68f6-5a07-41c2-acbe-58a2c98ebdd7" path="/var/lib/kubelet/pods/f18a68f6-5a07-41c2-acbe-58a2c98ebdd7/volumes"
Oct 29 22:14:38.442666 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:38.442634    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"]
Oct 29 22:14:38.443445 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:38.443406    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5" podUID="a437f4e2-dc10-41de-851a-ee488b08c6f4" containerName="istio-proxy" containerID="cri-o://edbdaac2be2e95304e218f52253c3354df3f1e301ecc6ae2fc886f2091923658" gracePeriod=30
Oct 29 22:14:38.450512 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:38.450487    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"]
Oct 29 22:14:38.450805 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:38.450738    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j" podUID="590ad9cd-3237-4783-8dea-82e67274b580" containerName="istio-proxy" containerID="cri-o://51b7485dab3d195f50fcae441fe893a75227dd05a3c12d77eb5b082648641ff5" gracePeriod=30
Oct 29 22:14:38.458349 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:38.458325    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"]
Oct 29 22:14:38.458502 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:38.458484    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc" podUID="b72eb074-2855-4dd2-bf0b-e4a42e92d8bb" containerName="istio-proxy" containerID="cri-o://d25824e90d88ebcedeea778ec0f75210f943a5a9ab3142969b7d5f26a4de63aa" gracePeriod=30
Oct 29 22:14:39.723032 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:14:39.722997    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:14:41.490061 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:41.490025    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"]
Oct 29 22:14:41.490499 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:41.490302    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2" podUID="03b2ca39-a1c7-45bf-8458-f23215d334b1" containerName="istio-proxy" containerID="cri-o://d6455e4a3c4bf2a3ecc64d4e53198a939f18bdd777da43a81925856a013ca70e" gracePeriod=30
Oct 29 22:14:41.506211 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:41.506180    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"]
Oct 29 22:14:41.506467 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:41.506415    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6" podUID="295e3d96-ff42-4be3-b4de-cd85ca4d8578" containerName="istio-proxy" containerID="cri-o://755a669af2f3c96b2ec3eec0dc1d9c2726cc8f9205c8cc4f080534a9a6684958" gracePeriod=30
Oct 29 22:14:41.511136 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:41.511114    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"]
Oct 29 22:14:41.511485 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:41.511456    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr" podUID="10c5ec48-f48a-4703-a82c-9a6995506125" containerName="istio-proxy" containerID="cri-o://c668a05f6f3d9dbc6215843071b014aba97a0397eae32e6077fb52f4bda89af1" gracePeriod=30
Oct 29 22:14:41.522721 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:41.522698    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"]
Oct 29 22:14:41.522888 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:41.522857    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6" podUID="64a76b6c-1542-4fb1-9730-7b770c452b56" containerName="istio-proxy" containerID="cri-o://fc31966e8412d3b25aade1df656c1c99dff20fed6545634e537352070118b61c" gracePeriod=30
Oct 29 22:14:42.409018 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:42.408978    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"]
Oct 29 22:14:42.409240 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:42.409215    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m" podUID="a3ee7c8e-f519-4574-a3be-f10d70926485" containerName="istio-proxy" containerID="cri-o://70ceada97cb370cd59ac50d3ebf826c8f79c5b98646f0ac0eace60ba02c5cdc1" gracePeriod=30
Oct 29 22:14:42.418754 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:42.418726    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"]
Oct 29 22:14:42.419008 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:42.418982    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2" podUID="71174a8c-e386-4670-942e-3bc750987080" containerName="istio-proxy" containerID="cri-o://0c1daee793a4f3895e641aab38ed7e988ab4dc2a0530cccf9d3942a518bd960e" gracePeriod=30
Oct 29 22:14:42.891132 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:42.891055    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:14:42.891132 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:42.891107    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2" podUID="71174a8c-e386-4670-942e-3bc750987080" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:14:43.572923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.572881    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:14:43.573044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.572952    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr" podUID="10c5ec48-f48a-4703-a82c-9a6995506125" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:14:43.742777 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.742752    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 22:14:43.763490 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.763423    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 22:14:43.790797 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.790768    4962 generic.go:356] "Generic (PLEG): container finished" podID="b72eb074-2855-4dd2-bf0b-e4a42e92d8bb" containerID="d25824e90d88ebcedeea778ec0f75210f943a5a9ab3142969b7d5f26a4de63aa" exitCode=0
Oct 29 22:14:43.790938 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.790843    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc" event={"ID":"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb","Type":"ContainerDied","Data":"d25824e90d88ebcedeea778ec0f75210f943a5a9ab3142969b7d5f26a4de63aa"}
Oct 29 22:14:43.792264 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.792238    4962 generic.go:356] "Generic (PLEG): container finished" podID="a437f4e2-dc10-41de-851a-ee488b08c6f4" containerID="edbdaac2be2e95304e218f52253c3354df3f1e301ecc6ae2fc886f2091923658" exitCode=0
Oct 29 22:14:43.792376 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.792327    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"
Oct 29 22:14:43.792486 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.792337    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5" event={"ID":"a437f4e2-dc10-41de-851a-ee488b08c6f4","Type":"ContainerDied","Data":"edbdaac2be2e95304e218f52253c3354df3f1e301ecc6ae2fc886f2091923658"}
Oct 29 22:14:43.792534 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.792519    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5" event={"ID":"a437f4e2-dc10-41de-851a-ee488b08c6f4","Type":"ContainerDied","Data":"34a9b53eba77b90358b87fe8e85198b3a1c4705ae6401fe42ce69e28a3eb000f"}
Oct 29 22:14:43.792669 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.792653    4962 scope.go:122] "RemoveContainer" containerID="edbdaac2be2e95304e218f52253c3354df3f1e301ecc6ae2fc886f2091923658"
Oct 29 22:14:43.793905 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.793887    4962 generic.go:356] "Generic (PLEG): container finished" podID="590ad9cd-3237-4783-8dea-82e67274b580" containerID="51b7485dab3d195f50fcae441fe893a75227dd05a3c12d77eb5b082648641ff5" exitCode=0
Oct 29 22:14:43.793977 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.793931    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j" event={"ID":"590ad9cd-3237-4783-8dea-82e67274b580","Type":"ContainerDied","Data":"51b7485dab3d195f50fcae441fe893a75227dd05a3c12d77eb5b082648641ff5"}
Oct 29 22:14:43.793977 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.793939    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"
Oct 29 22:14:43.793977 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.793958    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j" event={"ID":"590ad9cd-3237-4783-8dea-82e67274b580","Type":"ContainerDied","Data":"91531365cac88323750addb04f5b011dc313d5b61d846b8da9af74f0f0543a26"}
Oct 29 22:14:43.806419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.806394    4962 scope.go:122] "RemoveContainer" containerID="edbdaac2be2e95304e218f52253c3354df3f1e301ecc6ae2fc886f2091923658"
Oct 29 22:14:43.806764 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:14:43.806745    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edbdaac2be2e95304e218f52253c3354df3f1e301ecc6ae2fc886f2091923658\": container with ID starting with edbdaac2be2e95304e218f52253c3354df3f1e301ecc6ae2fc886f2091923658 not found: ID does not exist" containerID="edbdaac2be2e95304e218f52253c3354df3f1e301ecc6ae2fc886f2091923658"
Oct 29 22:14:43.806843 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.806770    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edbdaac2be2e95304e218f52253c3354df3f1e301ecc6ae2fc886f2091923658"} err="failed to get container status \"edbdaac2be2e95304e218f52253c3354df3f1e301ecc6ae2fc886f2091923658\": rpc error: code = NotFound desc = could not find container \"edbdaac2be2e95304e218f52253c3354df3f1e301ecc6ae2fc886f2091923658\": container with ID starting with edbdaac2be2e95304e218f52253c3354df3f1e301ecc6ae2fc886f2091923658 not found: ID does not exist"
Oct 29 22:14:43.806843 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.806790    4962 scope.go:122] "RemoveContainer" containerID="51b7485dab3d195f50fcae441fe893a75227dd05a3c12d77eb5b082648641ff5"
Oct 29 22:14:43.814172 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.813692    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-token\" (UniqueName: \"kubernetes.io/projected/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-token\") pod \"a437f4e2-dc10-41de-851a-ee488b08c6f4\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") "
Oct 29 22:14:43.814172 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.813780    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-workload-socket\") pod \"a437f4e2-dc10-41de-851a-ee488b08c6f4\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") "
Oct 29 22:14:43.814172 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.813846    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/590ad9cd-3237-4783-8dea-82e67274b580-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590ad9cd-3237-4783-8dea-82e67274b580-istiod-ca-cert\") pod \"590ad9cd-3237-4783-8dea-82e67274b580\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") "
Oct 29 22:14:43.814172 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.813893    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/590ad9cd-3237-4783-8dea-82e67274b580-istio-token\" (UniqueName: \"kubernetes.io/projected/590ad9cd-3237-4783-8dea-82e67274b580-istio-token\") pod \"590ad9cd-3237-4783-8dea-82e67274b580\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") "
Oct 29 22:14:43.814172 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.813963    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/590ad9cd-3237-4783-8dea-82e67274b580-kube-api-access-mhwwd\" (UniqueName: \"kubernetes.io/projected/590ad9cd-3237-4783-8dea-82e67274b580-kube-api-access-mhwwd\") pod \"590ad9cd-3237-4783-8dea-82e67274b580\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") "
Oct 29 22:14:43.814172 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.813993    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-podinfo\") pod \"a437f4e2-dc10-41de-851a-ee488b08c6f4\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") "
Oct 29 22:14:43.814172 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.814039    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/590ad9cd-3237-4783-8dea-82e67274b580-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590ad9cd-3237-4783-8dea-82e67274b580-istio-podinfo\") pod \"590ad9cd-3237-4783-8dea-82e67274b580\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") "
Oct 29 22:14:43.814172 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.814075    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-data\") pod \"a437f4e2-dc10-41de-851a-ee488b08c6f4\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") "
Oct 29 22:14:43.814172 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.814112    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-workload-socket\") pod \"590ad9cd-3237-4783-8dea-82e67274b580\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") "
Oct 29 22:14:43.814172 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.814178    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-credential-socket\") pod \"a437f4e2-dc10-41de-851a-ee488b08c6f4\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") "
Oct 29 22:14:43.814956 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.814211    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-workload-certs\") pod \"590ad9cd-3237-4783-8dea-82e67274b580\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") "
Oct 29 22:14:43.814956 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.814248    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-envoy\") pod \"a437f4e2-dc10-41de-851a-ee488b08c6f4\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") "
Oct 29 22:14:43.814956 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.814355    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-istio-envoy\") pod \"590ad9cd-3237-4783-8dea-82e67274b580\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") "
Oct 29 22:14:43.814956 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.814383    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-credential-socket\") pod \"590ad9cd-3237-4783-8dea-82e67274b580\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") "
Oct 29 22:14:43.814956 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.814408    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a437f4e2-dc10-41de-851a-ee488b08c6f4-kube-api-access-pw4lq\" (UniqueName: \"kubernetes.io/projected/a437f4e2-dc10-41de-851a-ee488b08c6f4-kube-api-access-pw4lq\") pod \"a437f4e2-dc10-41de-851a-ee488b08c6f4\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") "
Oct 29 22:14:43.814956 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.814454    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-workload-certs\") pod \"a437f4e2-dc10-41de-851a-ee488b08c6f4\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") "
Oct 29 22:14:43.814956 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.814488    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a437f4e2-dc10-41de-851a-ee488b08c6f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a437f4e2-dc10-41de-851a-ee488b08c6f4-istiod-ca-cert\") pod \"a437f4e2-dc10-41de-851a-ee488b08c6f4\" (UID: \"a437f4e2-dc10-41de-851a-ee488b08c6f4\") "
Oct 29 22:14:43.814956 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.814552    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-istio-data\") pod \"590ad9cd-3237-4783-8dea-82e67274b580\" (UID: \"590ad9cd-3237-4783-8dea-82e67274b580\") "
Oct 29 22:14:43.815888 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.815391    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-workload-socket" pod "a437f4e2-dc10-41de-851a-ee488b08c6f4" (UID: "a437f4e2-dc10-41de-851a-ee488b08c6f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.815888 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.815487    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-credential-socket" pod "a437f4e2-dc10-41de-851a-ee488b08c6f4" (UID: "a437f4e2-dc10-41de-851a-ee488b08c6f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.815888 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.815760    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-workload-certs" pod "590ad9cd-3237-4783-8dea-82e67274b580" (UID: "590ad9cd-3237-4783-8dea-82e67274b580"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.816517 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.816329    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-data" pod "a437f4e2-dc10-41de-851a-ee488b08c6f4" (UID: "a437f4e2-dc10-41de-851a-ee488b08c6f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.816725 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.816676    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-workload-socket" pod "590ad9cd-3237-4783-8dea-82e67274b580" (UID: "590ad9cd-3237-4783-8dea-82e67274b580"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.817251 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.817228    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-credential-socket" pod "590ad9cd-3237-4783-8dea-82e67274b580" (UID: "590ad9cd-3237-4783-8dea-82e67274b580"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.817655 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.817633    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-workload-certs" pod "a437f4e2-dc10-41de-851a-ee488b08c6f4" (UID: "a437f4e2-dc10-41de-851a-ee488b08c6f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.818075 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.818056    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a437f4e2-dc10-41de-851a-ee488b08c6f4-istiod-ca-cert" pod "a437f4e2-dc10-41de-851a-ee488b08c6f4" (UID: "a437f4e2-dc10-41de-851a-ee488b08c6f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:43.818263 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.818243    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-token" pod "a437f4e2-dc10-41de-851a-ee488b08c6f4" (UID: "a437f4e2-dc10-41de-851a-ee488b08c6f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:43.818572 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.818534    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/590ad9cd-3237-4783-8dea-82e67274b580-istiod-ca-cert" pod "590ad9cd-3237-4783-8dea-82e67274b580" (UID: "590ad9cd-3237-4783-8dea-82e67274b580"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:43.818817 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.818796    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/590ad9cd-3237-4783-8dea-82e67274b580-istio-podinfo" pod "590ad9cd-3237-4783-8dea-82e67274b580" (UID: "590ad9cd-3237-4783-8dea-82e67274b580"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:43.819559 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.819515    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-istio-data" pod "590ad9cd-3237-4783-8dea-82e67274b580" (UID: "590ad9cd-3237-4783-8dea-82e67274b580"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.819947 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.819921    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-envoy" pod "a437f4e2-dc10-41de-851a-ee488b08c6f4" (UID: "a437f4e2-dc10-41de-851a-ee488b08c6f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.820039 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.819942    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590ad9cd-3237-4783-8dea-82e67274b580-istio-token" pod "590ad9cd-3237-4783-8dea-82e67274b580" (UID: "590ad9cd-3237-4783-8dea-82e67274b580"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:43.821577 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.821560    4962 scope.go:122] "RemoveContainer" containerID="51b7485dab3d195f50fcae441fe893a75227dd05a3c12d77eb5b082648641ff5"
Oct 29 22:14:43.821920 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:14:43.821889    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51b7485dab3d195f50fcae441fe893a75227dd05a3c12d77eb5b082648641ff5\": container with ID starting with 51b7485dab3d195f50fcae441fe893a75227dd05a3c12d77eb5b082648641ff5 not found: ID does not exist" containerID="51b7485dab3d195f50fcae441fe893a75227dd05a3c12d77eb5b082648641ff5"
Oct 29 22:14:43.821982 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.821926    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51b7485dab3d195f50fcae441fe893a75227dd05a3c12d77eb5b082648641ff5"} err="failed to get container status \"51b7485dab3d195f50fcae441fe893a75227dd05a3c12d77eb5b082648641ff5\": rpc error: code = NotFound desc = could not find container \"51b7485dab3d195f50fcae441fe893a75227dd05a3c12d77eb5b082648641ff5\": container with ID starting with 51b7485dab3d195f50fcae441fe893a75227dd05a3c12d77eb5b082648641ff5 not found: ID does not exist"
Oct 29 22:14:43.822740 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.822718    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-istio-envoy" pod "590ad9cd-3237-4783-8dea-82e67274b580" (UID: "590ad9cd-3237-4783-8dea-82e67274b580"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.822827 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.822753    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a437f4e2-dc10-41de-851a-ee488b08c6f4-kube-api-access-pw4lq" pod "a437f4e2-dc10-41de-851a-ee488b08c6f4" (UID: "a437f4e2-dc10-41de-851a-ee488b08c6f4"). InnerVolumeSpecName "kube-api-access-pw4lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:43.822827 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.822809    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-podinfo" pod "a437f4e2-dc10-41de-851a-ee488b08c6f4" (UID: "a437f4e2-dc10-41de-851a-ee488b08c6f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:43.822927 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.822899    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590ad9cd-3237-4783-8dea-82e67274b580-kube-api-access-mhwwd" pod "590ad9cd-3237-4783-8dea-82e67274b580" (UID: "590ad9cd-3237-4783-8dea-82e67274b580"). InnerVolumeSpecName "kube-api-access-mhwwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:43.829130 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.829100    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 22:14:43.915917 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.915886    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-podinfo\") pod \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") "
Oct 29 22:14:43.916277 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.915981    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-data\") pod \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") "
Oct 29 22:14:43.916277 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.916085    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-workload-certs\") pod \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") "
Oct 29 22:14:43.916277 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.916109    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-token\" (UniqueName: \"kubernetes.io/projected/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-token\") pod \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") "
Oct 29 22:14:43.916277 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.916134    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istiod-ca-cert\") pod \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") "
Oct 29 22:14:43.916277 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.916159    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-credential-socket\") pod \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") "
Oct 29 22:14:43.916277 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.916227    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-kube-api-access-knkff\" (UniqueName: \"kubernetes.io/projected/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-kube-api-access-knkff\") pod \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") "
Oct 29 22:14:43.916529 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.916316    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-data" pod "b72eb074-2855-4dd2-bf0b-e4a42e92d8bb" (UID: "b72eb074-2855-4dd2-bf0b-e4a42e92d8bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.916529 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.916326    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-envoy\") pod \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") "
Oct 29 22:14:43.916529 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.916423    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-workload-socket\") pod \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\" (UID: \"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb\") "
Oct 29 22:14:43.916664 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.916615    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istiod-ca-cert" pod "b72eb074-2855-4dd2-bf0b-e4a42e92d8bb" (UID: "b72eb074-2855-4dd2-bf0b-e4a42e92d8bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:43.916861 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.916840    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-credential-socket" pod "b72eb074-2855-4dd2-bf0b-e4a42e92d8bb" (UID: "b72eb074-2855-4dd2-bf0b-e4a42e92d8bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.916979 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.916951    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-workload-certs" pod "b72eb074-2855-4dd2-bf0b-e4a42e92d8bb" (UID: "b72eb074-2855-4dd2-bf0b-e4a42e92d8bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.917039    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-workload-socket" pod "b72eb074-2855-4dd2-bf0b-e4a42e92d8bb" (UID: "b72eb074-2855-4dd2-bf0b-e4a42e92d8bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.917173    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.917186    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.917205    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.917224    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.917238    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw4lq\" (UniqueName: \"kubernetes.io/projected/a437f4e2-dc10-41de-851a-ee488b08c6f4-kube-api-access-pw4lq\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.917252    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.917265    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.917278    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.917308    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a437f4e2-dc10-41de-851a-ee488b08c6f4-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.917319    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.917330    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.917342    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.917356    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.917369    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/590ad9cd-3237-4783-8dea-82e67274b580-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.917381    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/590ad9cd-3237-4783-8dea-82e67274b580-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.917393    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhwwd\" (UniqueName: \"kubernetes.io/projected/590ad9cd-3237-4783-8dea-82e67274b580-kube-api-access-mhwwd\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.917407    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.917420    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/590ad9cd-3237-4783-8dea-82e67274b580-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.917433    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.917447    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.917460    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.917473    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a437f4e2-dc10-41de-851a-ee488b08c6f4-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.917487    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/590ad9cd-3237-4783-8dea-82e67274b580-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.918655    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-podinfo" pod "b72eb074-2855-4dd2-bf0b-e4a42e92d8bb" (UID: "b72eb074-2855-4dd2-bf0b-e4a42e92d8bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.918908    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-envoy" pod "b72eb074-2855-4dd2-bf0b-e4a42e92d8bb" (UID: "b72eb074-2855-4dd2-bf0b-e4a42e92d8bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.918961    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-token" pod "b72eb074-2855-4dd2-bf0b-e4a42e92d8bb" (UID: "b72eb074-2855-4dd2-bf0b-e4a42e92d8bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:43.919401 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.919023    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-kube-api-access-knkff" pod "b72eb074-2855-4dd2-bf0b-e4a42e92d8bb" (UID: "b72eb074-2855-4dd2-bf0b-e4a42e92d8bb"). InnerVolumeSpecName "kube-api-access-knkff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:43.958370 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.958331    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:14:43.958534 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:43.958407    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6" podUID="64a76b6c-1542-4fb1-9730-7b770c452b56" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:14:44.018658 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:44.018588    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.018658 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:44.018613    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.018658 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:44.018623    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.018658 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:44.018632    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knkff\" (UniqueName: \"kubernetes.io/projected/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb-kube-api-access-knkff\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:44.145868 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:44.145838    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"]
Oct 29 22:14:44.152674 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:44.152643    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-mzpw5"]
Oct 29 22:14:44.186069 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:44.186027    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"]
Oct 29 22:14:44.189508 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:44.189477    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-4qm6j"]
Oct 29 22:14:44.802370 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:44.802261    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"
Oct 29 22:14:44.802370 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:44.802261    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc" event={"ID":"b72eb074-2855-4dd2-bf0b-e4a42e92d8bb","Type":"ContainerDied","Data":"c675f7f724204b29e627a57d9fc5250553f253e612491d9ed83cbedb4b3d801b"}
Oct 29 22:14:44.802581 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:44.802428    4962 scope.go:122] "RemoveContainer" containerID="d25824e90d88ebcedeea778ec0f75210f943a5a9ab3142969b7d5f26a4de63aa"
Oct 29 22:14:44.854755 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:44.854728    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"]
Oct 29 22:14:44.864502 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:44.864480    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a940bc-openshift-default-6d694cb79f-nvzzc"]
Oct 29 22:14:45.058587 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:45.058522    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="590ad9cd-3237-4783-8dea-82e67274b580" path="/var/lib/kubelet/pods/590ad9cd-3237-4783-8dea-82e67274b580/volumes"
Oct 29 22:14:45.059118 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:45.058994    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a437f4e2-dc10-41de-851a-ee488b08c6f4" path="/var/lib/kubelet/pods/a437f4e2-dc10-41de-851a-ee488b08c6f4/volumes"
Oct 29 22:14:45.059447 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:45.059430    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b72eb074-2855-4dd2-bf0b-e4a42e92d8bb" path="/var/lib/kubelet/pods/b72eb074-2855-4dd2-bf0b-e4a42e92d8bb/volumes"
Oct 29 22:14:46.040891 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.040855    4962 patch_prober.go:29] interesting pod/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:14:46.041063 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.040921    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6" podUID="295e3d96-ff42-4be3-b4de-cd85ca4d8578" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:14:46.839198 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.839164    4962 generic.go:356] "Generic (PLEG): container finished" podID="03b2ca39-a1c7-45bf-8458-f23215d334b1" containerID="d6455e4a3c4bf2a3ecc64d4e53198a939f18bdd777da43a81925856a013ca70e" exitCode=0
Oct 29 22:14:46.839642 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.839349    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2" event={"ID":"03b2ca39-a1c7-45bf-8458-f23215d334b1","Type":"ContainerDied","Data":"d6455e4a3c4bf2a3ecc64d4e53198a939f18bdd777da43a81925856a013ca70e"}
Oct 29 22:14:46.839642 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.839533    4962 scope.go:122] "RemoveContainer" containerID="d54981ad8aa1ebada11c30c0ac5a4d1cf6117480f18fec0001c3f840f139970a"
Oct 29 22:14:46.839771 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.839719    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:14:46.857522 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.851452    4962 generic.go:356] "Generic (PLEG): container finished" podID="10c5ec48-f48a-4703-a82c-9a6995506125" containerID="c668a05f6f3d9dbc6215843071b014aba97a0397eae32e6077fb52f4bda89af1" exitCode=0
Oct 29 22:14:46.857522 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.851690    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr" event={"ID":"10c5ec48-f48a-4703-a82c-9a6995506125","Type":"ContainerDied","Data":"c668a05f6f3d9dbc6215843071b014aba97a0397eae32e6077fb52f4bda89af1"}
Oct 29 22:14:46.857522 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.855539    4962 generic.go:356] "Generic (PLEG): container finished" podID="64a76b6c-1542-4fb1-9730-7b770c452b56" containerID="fc31966e8412d3b25aade1df656c1c99dff20fed6545634e537352070118b61c" exitCode=0
Oct 29 22:14:46.857522 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.855625    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6" event={"ID":"64a76b6c-1542-4fb1-9730-7b770c452b56","Type":"ContainerDied","Data":"fc31966e8412d3b25aade1df656c1c99dff20fed6545634e537352070118b61c"}
Oct 29 22:14:46.857522 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.857080    4962 generic.go:356] "Generic (PLEG): container finished" podID="295e3d96-ff42-4be3-b4de-cd85ca4d8578" containerID="755a669af2f3c96b2ec3eec0dc1d9c2726cc8f9205c8cc4f080534a9a6684958" exitCode=0
Oct 29 22:14:46.857522 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.857126    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6" event={"ID":"295e3d96-ff42-4be3-b4de-cd85ca4d8578","Type":"ContainerDied","Data":"755a669af2f3c96b2ec3eec0dc1d9c2726cc8f9205c8cc4f080534a9a6684958"}
Oct 29 22:14:46.857522 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.857148    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"
Oct 29 22:14:46.857522 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.857163    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6" event={"ID":"295e3d96-ff42-4be3-b4de-cd85ca4d8578","Type":"ContainerDied","Data":"a8f80e2cfc0567a627f4e74c7f93478e7b2d094e7ca7408ca60923b63fcd8a9d"}
Oct 29 22:14:46.896403 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.896103    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 22:14:46.899658 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.899627    4962 scope.go:122] "RemoveContainer" containerID="755a669af2f3c96b2ec3eec0dc1d9c2726cc8f9205c8cc4f080534a9a6684958"
Oct 29 22:14:46.920920 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.920900    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 22:14:46.922230 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.922211    4962 scope.go:122] "RemoveContainer" containerID="755a669af2f3c96b2ec3eec0dc1d9c2726cc8f9205c8cc4f080534a9a6684958"
Oct 29 22:14:46.922566 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:14:46.922547    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"755a669af2f3c96b2ec3eec0dc1d9c2726cc8f9205c8cc4f080534a9a6684958\": container with ID starting with 755a669af2f3c96b2ec3eec0dc1d9c2726cc8f9205c8cc4f080534a9a6684958 not found: ID does not exist" containerID="755a669af2f3c96b2ec3eec0dc1d9c2726cc8f9205c8cc4f080534a9a6684958"
Oct 29 22:14:46.922653 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.922580    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"755a669af2f3c96b2ec3eec0dc1d9c2726cc8f9205c8cc4f080534a9a6684958"} err="failed to get container status \"755a669af2f3c96b2ec3eec0dc1d9c2726cc8f9205c8cc4f080534a9a6684958\": rpc error: code = NotFound desc = could not find container \"755a669af2f3c96b2ec3eec0dc1d9c2726cc8f9205c8cc4f080534a9a6684958\": container with ID starting with 755a669af2f3c96b2ec3eec0dc1d9c2726cc8f9205c8cc4f080534a9a6684958 not found: ID does not exist"
Oct 29 22:14:46.923498 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.923473    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:14:46.954269 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.954096    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03b2ca39-a1c7-45bf-8458-f23215d334b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b2ca39-a1c7-45bf-8458-f23215d334b1-istiod-ca-cert\") pod \"03b2ca39-a1c7-45bf-8458-f23215d334b1\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") "
Oct 29 22:14:46.954269 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.954137    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-istio-data\") pod \"10c5ec48-f48a-4703-a82c-9a6995506125\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") "
Oct 29 22:14:46.954269 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.954169    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10c5ec48-f48a-4703-a82c-9a6995506125-kube-api-access-lbhq7\" (UniqueName: \"kubernetes.io/projected/10c5ec48-f48a-4703-a82c-9a6995506125-kube-api-access-lbhq7\") pod \"10c5ec48-f48a-4703-a82c-9a6995506125\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") "
Oct 29 22:14:46.954269 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.954196    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-envoy\") pod \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") "
Oct 29 22:14:46.954269 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.954216    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istiod-ca-cert\") pod \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") "
Oct 29 22:14:46.954269 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.954241    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-workload-certs\") pod \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") "
Oct 29 22:14:46.954269 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.954260    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-credential-socket\") pod \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") "
Oct 29 22:14:46.954630 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.954318    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-podinfo\") pod \"03b2ca39-a1c7-45bf-8458-f23215d334b1\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") "
Oct 29 22:14:46.954630 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.954363    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-envoy\") pod \"03b2ca39-a1c7-45bf-8458-f23215d334b1\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") "
Oct 29 22:14:46.954630 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.954395    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-envoy\") pod \"64a76b6c-1542-4fb1-9730-7b770c452b56\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") "
Oct 29 22:14:46.954630 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.954420    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-workload-socket\") pod \"03b2ca39-a1c7-45bf-8458-f23215d334b1\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") "
Oct 29 22:14:46.954630 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.954449    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10c5ec48-f48a-4703-a82c-9a6995506125-istio-token\" (UniqueName: \"kubernetes.io/projected/10c5ec48-f48a-4703-a82c-9a6995506125-istio-token\") pod \"10c5ec48-f48a-4703-a82c-9a6995506125\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") "
Oct 29 22:14:46.954630 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.954469    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-workload-certs\") pod \"10c5ec48-f48a-4703-a82c-9a6995506125\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") "
Oct 29 22:14:46.954630 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.954555    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b2ca39-a1c7-45bf-8458-f23215d334b1-istiod-ca-cert" pod "03b2ca39-a1c7-45bf-8458-f23215d334b1" (UID: "03b2ca39-a1c7-45bf-8458-f23215d334b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:46.954856 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.954714    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-istio-data" pod "10c5ec48-f48a-4703-a82c-9a6995506125" (UID: "10c5ec48-f48a-4703-a82c-9a6995506125"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.954856 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.954733    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-workload-certs" pod "10c5ec48-f48a-4703-a82c-9a6995506125" (UID: "10c5ec48-f48a-4703-a82c-9a6995506125"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.955014 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.954989    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-workload-socket" pod "03b2ca39-a1c7-45bf-8458-f23215d334b1" (UID: "03b2ca39-a1c7-45bf-8458-f23215d334b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.955429    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-workload-certs" pod "295e3d96-ff42-4be3-b4de-cd85ca4d8578" (UID: "295e3d96-ff42-4be3-b4de-cd85ca4d8578"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.955839    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istiod-ca-cert" pod "295e3d96-ff42-4be3-b4de-cd85ca4d8578" (UID: "295e3d96-ff42-4be3-b4de-cd85ca4d8578"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.955892    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64a76b6c-1542-4fb1-9730-7b770c452b56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64a76b6c-1542-4fb1-9730-7b770c452b56-istiod-ca-cert\") pod \"64a76b6c-1542-4fb1-9730-7b770c452b56\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") "
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.955947    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-credential-socket\") pod \"10c5ec48-f48a-4703-a82c-9a6995506125\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") "
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.956006    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-token\" (UniqueName: \"kubernetes.io/projected/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-token\") pod \"03b2ca39-a1c7-45bf-8458-f23215d334b1\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") "
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.956048    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-data\") pod \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") "
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.956109    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10c5ec48-f48a-4703-a82c-9a6995506125-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c5ec48-f48a-4703-a82c-9a6995506125-istio-podinfo\") pod \"10c5ec48-f48a-4703-a82c-9a6995506125\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") "
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.956139    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-token\" (UniqueName: \"kubernetes.io/projected/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-token\") pod \"64a76b6c-1542-4fb1-9730-7b770c452b56\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") "
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.956176    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03b2ca39-a1c7-45bf-8458-f23215d334b1-kube-api-access-lh82l\" (UniqueName: \"kubernetes.io/projected/03b2ca39-a1c7-45bf-8458-f23215d334b1-kube-api-access-lh82l\") pod \"03b2ca39-a1c7-45bf-8458-f23215d334b1\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") "
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.956206    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/295e3d96-ff42-4be3-b4de-cd85ca4d8578-kube-api-access-wf79q\" (UniqueName: \"kubernetes.io/projected/295e3d96-ff42-4be3-b4de-cd85ca4d8578-kube-api-access-wf79q\") pod \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") "
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.956271    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-token\" (UniqueName: \"kubernetes.io/projected/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-token\") pod \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") "
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.956335    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10c5ec48-f48a-4703-a82c-9a6995506125-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c5ec48-f48a-4703-a82c-9a6995506125-istiod-ca-cert\") pod \"10c5ec48-f48a-4703-a82c-9a6995506125\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") "
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.956422    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-credential-socket\") pod \"64a76b6c-1542-4fb1-9730-7b770c452b56\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") "
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.956465    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-workload-socket\") pod \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") "
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.956499    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a76b6c-1542-4fb1-9730-7b770c452b56-kube-api-access-xpwj9\" (UniqueName: \"kubernetes.io/projected/64a76b6c-1542-4fb1-9730-7b770c452b56-kube-api-access-xpwj9\") pod \"64a76b6c-1542-4fb1-9730-7b770c452b56\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") "
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.956538    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-istio-envoy\") pod \"10c5ec48-f48a-4703-a82c-9a6995506125\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") "
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.956588    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-podinfo\") pod \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\" (UID: \"295e3d96-ff42-4be3-b4de-cd85ca4d8578\") "
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.956619    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-data\") pod \"03b2ca39-a1c7-45bf-8458-f23215d334b1\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") "
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.956664    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-workload-socket\") pod \"10c5ec48-f48a-4703-a82c-9a6995506125\" (UID: \"10c5ec48-f48a-4703-a82c-9a6995506125\") "
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.956699    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-credential-socket\") pod \"03b2ca39-a1c7-45bf-8458-f23215d334b1\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") "
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.956735    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-workload-certs\") pod \"03b2ca39-a1c7-45bf-8458-f23215d334b1\" (UID: \"03b2ca39-a1c7-45bf-8458-f23215d334b1\") "
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.956806    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-data\") pod \"64a76b6c-1542-4fb1-9730-7b770c452b56\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") "
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.956844    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-workload-certs\") pod \"64a76b6c-1542-4fb1-9730-7b770c452b56\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") "
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.957000    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-podinfo\") pod \"64a76b6c-1542-4fb1-9730-7b770c452b56\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") "
Oct 29 22:14:46.957419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.957047    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-workload-socket\") pod \"64a76b6c-1542-4fb1-9730-7b770c452b56\" (UID: \"64a76b6c-1542-4fb1-9730-7b770c452b56\") "
Oct 29 22:14:46.960103 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.957686    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c5ec48-f48a-4703-a82c-9a6995506125-kube-api-access-lbhq7" pod "10c5ec48-f48a-4703-a82c-9a6995506125" (UID: "10c5ec48-f48a-4703-a82c-9a6995506125"). InnerVolumeSpecName "kube-api-access-lbhq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:46.960103 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.957726    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c5ec48-f48a-4703-a82c-9a6995506125-istio-token" pod "10c5ec48-f48a-4703-a82c-9a6995506125" (UID: "10c5ec48-f48a-4703-a82c-9a6995506125"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:46.960103 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.957982    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-credential-socket" pod "64a76b6c-1542-4fb1-9730-7b770c452b56" (UID: "64a76b6c-1542-4fb1-9730-7b770c452b56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.960103 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.958236    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a76b6c-1542-4fb1-9730-7b770c452b56-istiod-ca-cert" pod "64a76b6c-1542-4fb1-9730-7b770c452b56" (UID: "64a76b6c-1542-4fb1-9730-7b770c452b56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:46.960103 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.958460    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-envoy" pod "03b2ca39-a1c7-45bf-8458-f23215d334b1" (UID: "03b2ca39-a1c7-45bf-8458-f23215d334b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.960103 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.958479    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-podinfo" pod "03b2ca39-a1c7-45bf-8458-f23215d334b1" (UID: "03b2ca39-a1c7-45bf-8458-f23215d334b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:46.960103 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.958924    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-credential-socket" pod "03b2ca39-a1c7-45bf-8458-f23215d334b1" (UID: "03b2ca39-a1c7-45bf-8458-f23215d334b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.960103 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.959081    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-workload-certs" pod "03b2ca39-a1c7-45bf-8458-f23215d334b1" (UID: "03b2ca39-a1c7-45bf-8458-f23215d334b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.960103 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.959233    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-data" pod "64a76b6c-1542-4fb1-9730-7b770c452b56" (UID: "64a76b6c-1542-4fb1-9730-7b770c452b56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.960103 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.959403    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-workload-certs" pod "64a76b6c-1542-4fb1-9730-7b770c452b56" (UID: "64a76b6c-1542-4fb1-9730-7b770c452b56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.960965 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.960803    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-credential-socket" pod "295e3d96-ff42-4be3-b4de-cd85ca4d8578" (UID: "295e3d96-ff42-4be3-b4de-cd85ca4d8578"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.961351 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.961090    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-data" pod "295e3d96-ff42-4be3-b4de-cd85ca4d8578" (UID: "295e3d96-ff42-4be3-b4de-cd85ca4d8578"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.965551 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.964483    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-workload-socket" pod "10c5ec48-f48a-4703-a82c-9a6995506125" (UID: "10c5ec48-f48a-4703-a82c-9a6995506125"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.965551 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.964677    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-workload-socket" pod "295e3d96-ff42-4be3-b4de-cd85ca4d8578" (UID: "295e3d96-ff42-4be3-b4de-cd85ca4d8578"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.965551 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.964999    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-credential-socket" pod "10c5ec48-f48a-4703-a82c-9a6995506125" (UID: "10c5ec48-f48a-4703-a82c-9a6995506125"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.967189 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.967164    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-workload-socket" pod "64a76b6c-1542-4fb1-9730-7b770c452b56" (UID: "64a76b6c-1542-4fb1-9730-7b770c452b56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.967388 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.967308    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b2ca39-a1c7-45bf-8458-f23215d334b1-kube-api-access-lh82l" pod "03b2ca39-a1c7-45bf-8458-f23215d334b1" (UID: "03b2ca39-a1c7-45bf-8458-f23215d334b1"). InnerVolumeSpecName "kube-api-access-lh82l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:46.967462 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.967435    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-data" pod "03b2ca39-a1c7-45bf-8458-f23215d334b1" (UID: "03b2ca39-a1c7-45bf-8458-f23215d334b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.967931 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.967899    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.967931 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.967926    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh82l\" (UniqueName: \"kubernetes.io/projected/03b2ca39-a1c7-45bf-8458-f23215d334b1-kube-api-access-lh82l\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.968091 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.967943    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.968091 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.967956    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.968091 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.967969    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.968091 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.967983    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.968091 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.967996    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.968091 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.968009    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.968091 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.968058    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.968091 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.968072    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.968091 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.968086    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.968648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.968099    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03b2ca39-a1c7-45bf-8458-f23215d334b1-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.968648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.968113    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.968648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.968127    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbhq7\" (UniqueName: \"kubernetes.io/projected/10c5ec48-f48a-4703-a82c-9a6995506125-kube-api-access-lbhq7\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.968648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.968140    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.968648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.968154    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.968648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.968166    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.968648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.968180    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.968648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.968197    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.968648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.968210    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03b2ca39-a1c7-45bf-8458-f23215d334b1-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.968648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.968223    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c5ec48-f48a-4703-a82c-9a6995506125-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.968648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.968238    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.968648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.968250    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64a76b6c-1542-4fb1-9730-7b770c452b56-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.968648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.968264    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:46.968648 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.968247    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10c5ec48-f48a-4703-a82c-9a6995506125-istiod-ca-cert" pod "10c5ec48-f48a-4703-a82c-9a6995506125" (UID: "10c5ec48-f48a-4703-a82c-9a6995506125"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:46.970808 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.970762    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-podinfo" pod "295e3d96-ff42-4be3-b4de-cd85ca4d8578" (UID: "295e3d96-ff42-4be3-b4de-cd85ca4d8578"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:46.970939 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.970920    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295e3d96-ff42-4be3-b4de-cd85ca4d8578-kube-api-access-wf79q" pod "295e3d96-ff42-4be3-b4de-cd85ca4d8578" (UID: "295e3d96-ff42-4be3-b4de-cd85ca4d8578"). InnerVolumeSpecName "kube-api-access-wf79q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:46.972707 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.972568    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10c5ec48-f48a-4703-a82c-9a6995506125-istio-podinfo" pod "10c5ec48-f48a-4703-a82c-9a6995506125" (UID: "10c5ec48-f48a-4703-a82c-9a6995506125"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:46.972707 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.972590    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-envoy" pod "295e3d96-ff42-4be3-b4de-cd85ca4d8578" (UID: "295e3d96-ff42-4be3-b4de-cd85ca4d8578"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.972707 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.972597    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a76b6c-1542-4fb1-9730-7b770c452b56-kube-api-access-xpwj9" pod "64a76b6c-1542-4fb1-9730-7b770c452b56" (UID: "64a76b6c-1542-4fb1-9730-7b770c452b56"). InnerVolumeSpecName "kube-api-access-xpwj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:46.972707 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.972662    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-podinfo" pod "64a76b6c-1542-4fb1-9730-7b770c452b56" (UID: "64a76b6c-1542-4fb1-9730-7b770c452b56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:46.972707 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.972672    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-token" pod "64a76b6c-1542-4fb1-9730-7b770c452b56" (UID: "64a76b6c-1542-4fb1-9730-7b770c452b56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:46.972707 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.972675    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-token" pod "03b2ca39-a1c7-45bf-8458-f23215d334b1" (UID: "03b2ca39-a1c7-45bf-8458-f23215d334b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:46.972989 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.972708    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-token" pod "295e3d96-ff42-4be3-b4de-cd85ca4d8578" (UID: "295e3d96-ff42-4be3-b4de-cd85ca4d8578"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:46.973520 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.973497    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-envoy" pod "64a76b6c-1542-4fb1-9730-7b770c452b56" (UID: "64a76b6c-1542-4fb1-9730-7b770c452b56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:46.973786 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:46.973767    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-istio-envoy" pod "10c5ec48-f48a-4703-a82c-9a6995506125" (UID: "10c5ec48-f48a-4703-a82c-9a6995506125"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.077605 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.077523    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03b2ca39-a1c7-45bf-8458-f23215d334b1-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.077605 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.077561    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c5ec48-f48a-4703-a82c-9a6995506125-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.077605 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.077574    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.077923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.077611    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf79q\" (UniqueName: \"kubernetes.io/projected/295e3d96-ff42-4be3-b4de-cd85ca4d8578-kube-api-access-wf79q\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.077923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.077627    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.077923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.077640    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c5ec48-f48a-4703-a82c-9a6995506125-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.077923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.077653    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpwj9\" (UniqueName: \"kubernetes.io/projected/64a76b6c-1542-4fb1-9730-7b770c452b56-kube-api-access-xpwj9\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.077923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.077666    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c5ec48-f48a-4703-a82c-9a6995506125-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.077923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.077679    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.077923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.077692    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.077923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.077705    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/295e3d96-ff42-4be3-b4de-cd85ca4d8578-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.077923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.077717    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64a76b6c-1542-4fb1-9730-7b770c452b56-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.201419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.201390    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"]
Oct 29 22:14:47.203097 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.203067    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-7lpw6"]
Oct 29 22:14:47.697897 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.697842    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:14:47.739724 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.739700    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:14:47.786681 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.786650    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-credential-socket\") pod \"a3ee7c8e-f519-4574-a3be-f10d70926485\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") "
Oct 29 22:14:47.786839 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.786696    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-workload-certs\") pod \"a3ee7c8e-f519-4574-a3be-f10d70926485\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") "
Oct 29 22:14:47.786839 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.786748    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-workload-socket\") pod \"71174a8c-e386-4670-942e-3bc750987080\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") "
Oct 29 22:14:47.786839 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.786775    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-istio-envoy\") pod \"71174a8c-e386-4670-942e-3bc750987080\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") "
Oct 29 22:14:47.786839 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.786814    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71174a8c-e386-4670-942e-3bc750987080-istio-token\" (UniqueName: \"kubernetes.io/projected/71174a8c-e386-4670-942e-3bc750987080-istio-token\") pod \"71174a8c-e386-4670-942e-3bc750987080\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") "
Oct 29 22:14:47.787074 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.786897    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-podinfo\") pod \"a3ee7c8e-f519-4574-a3be-f10d70926485\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") "
Oct 29 22:14:47.787074 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.786935    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-credential-socket\") pod \"71174a8c-e386-4670-942e-3bc750987080\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") "
Oct 29 22:14:47.787074 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.786964    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71174a8c-e386-4670-942e-3bc750987080-kube-api-access-sp8ht\" (UniqueName: \"kubernetes.io/projected/71174a8c-e386-4670-942e-3bc750987080-kube-api-access-sp8ht\") pod \"71174a8c-e386-4670-942e-3bc750987080\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") "
Oct 29 22:14:47.787074 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.787001    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-credential-socket" pod "a3ee7c8e-f519-4574-a3be-f10d70926485" (UID: "a3ee7c8e-f519-4574-a3be-f10d70926485"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.787074 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.787036    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-data\") pod \"a3ee7c8e-f519-4574-a3be-f10d70926485\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") "
Oct 29 22:14:47.787074 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.787038    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-workload-socket" pod "71174a8c-e386-4670-942e-3bc750987080" (UID: "71174a8c-e386-4670-942e-3bc750987080"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.787074 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.787071    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-envoy\") pod \"a3ee7c8e-f519-4574-a3be-f10d70926485\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") "
Oct 29 22:14:47.787482 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.787202    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-workload-certs" pod "a3ee7c8e-f519-4574-a3be-f10d70926485" (UID: "a3ee7c8e-f519-4574-a3be-f10d70926485"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.787482 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.787463    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-data" pod "a3ee7c8e-f519-4574-a3be-f10d70926485" (UID: "a3ee7c8e-f519-4574-a3be-f10d70926485"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.787595 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.787474    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-credential-socket" pod "71174a8c-e386-4670-942e-3bc750987080" (UID: "71174a8c-e386-4670-942e-3bc750987080"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.787595 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.787513    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71174a8c-e386-4670-942e-3bc750987080-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71174a8c-e386-4670-942e-3bc750987080-istiod-ca-cert\") pod \"71174a8c-e386-4670-942e-3bc750987080\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") "
Oct 29 22:14:47.787595 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.787557    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71174a8c-e386-4670-942e-3bc750987080-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71174a8c-e386-4670-942e-3bc750987080-istio-podinfo\") pod \"71174a8c-e386-4670-942e-3bc750987080\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") "
Oct 29 22:14:47.787756 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.787593    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-workload-certs\") pod \"71174a8c-e386-4670-942e-3bc750987080\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") "
Oct 29 22:14:47.787756 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.787627    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-workload-socket\") pod \"a3ee7c8e-f519-4574-a3be-f10d70926485\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") "
Oct 29 22:14:47.787756 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.787716    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-token\" (UniqueName: \"kubernetes.io/projected/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-token\") pod \"a3ee7c8e-f519-4574-a3be-f10d70926485\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") "
Oct 29 22:14:47.787905 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.787776    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-istio-data\") pod \"71174a8c-e386-4670-942e-3bc750987080\" (UID: \"71174a8c-e386-4670-942e-3bc750987080\") "
Oct 29 22:14:47.787905 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.787808    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ee7c8e-f519-4574-a3be-f10d70926485-kube-api-access-rzgfb\" (UniqueName: \"kubernetes.io/projected/a3ee7c8e-f519-4574-a3be-f10d70926485-kube-api-access-rzgfb\") pod \"a3ee7c8e-f519-4574-a3be-f10d70926485\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") "
Oct 29 22:14:47.787905 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.787850    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3ee7c8e-f519-4574-a3be-f10d70926485-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ee7c8e-f519-4574-a3be-f10d70926485-istiod-ca-cert\") pod \"a3ee7c8e-f519-4574-a3be-f10d70926485\" (UID: \"a3ee7c8e-f519-4574-a3be-f10d70926485\") "
Oct 29 22:14:47.788881 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.788417    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-workload-certs" pod "71174a8c-e386-4670-942e-3bc750987080" (UID: "71174a8c-e386-4670-942e-3bc750987080"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.788881 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.788581    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.788881 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.788602    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.788881 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.788618    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.788881 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.788631    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.788881 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.788643    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.788881 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.788655    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.788881 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.788791    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-istio-data" pod "71174a8c-e386-4670-942e-3bc750987080" (UID: "71174a8c-e386-4670-942e-3bc750987080"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.789390 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.788915    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71174a8c-e386-4670-942e-3bc750987080-istiod-ca-cert" pod "71174a8c-e386-4670-942e-3bc750987080" (UID: "71174a8c-e386-4670-942e-3bc750987080"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:47.789390 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.788988    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-workload-socket" pod "a3ee7c8e-f519-4574-a3be-f10d70926485" (UID: "a3ee7c8e-f519-4574-a3be-f10d70926485"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.789390 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.788717    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3ee7c8e-f519-4574-a3be-f10d70926485-istiod-ca-cert" pod "a3ee7c8e-f519-4574-a3be-f10d70926485" (UID: "a3ee7c8e-f519-4574-a3be-f10d70926485"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:47.791533 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.791503    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-podinfo" pod "a3ee7c8e-f519-4574-a3be-f10d70926485" (UID: "a3ee7c8e-f519-4574-a3be-f10d70926485"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:47.792041 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.792006    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-token" pod "a3ee7c8e-f519-4574-a3be-f10d70926485" (UID: "a3ee7c8e-f519-4574-a3be-f10d70926485"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:47.792443 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.792416    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ee7c8e-f519-4574-a3be-f10d70926485-kube-api-access-rzgfb" pod "a3ee7c8e-f519-4574-a3be-f10d70926485" (UID: "a3ee7c8e-f519-4574-a3be-f10d70926485"). InnerVolumeSpecName "kube-api-access-rzgfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:47.793367 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.793319    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-envoy" pod "a3ee7c8e-f519-4574-a3be-f10d70926485" (UID: "a3ee7c8e-f519-4574-a3be-f10d70926485"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.793588 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.793547    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71174a8c-e386-4670-942e-3bc750987080-istio-podinfo" pod "71174a8c-e386-4670-942e-3bc750987080" (UID: "71174a8c-e386-4670-942e-3bc750987080"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:14:47.793588 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.793573    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71174a8c-e386-4670-942e-3bc750987080-istio-token" pod "71174a8c-e386-4670-942e-3bc750987080" (UID: "71174a8c-e386-4670-942e-3bc750987080"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:47.793982 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.793957    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-istio-envoy" pod "71174a8c-e386-4670-942e-3bc750987080" (UID: "71174a8c-e386-4670-942e-3bc750987080"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:14:47.794069 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.793993    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71174a8c-e386-4670-942e-3bc750987080-kube-api-access-sp8ht" pod "71174a8c-e386-4670-942e-3bc750987080" (UID: "71174a8c-e386-4670-942e-3bc750987080"). InnerVolumeSpecName "kube-api-access-sp8ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:47.864878 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.864843    4962 generic.go:356] "Generic (PLEG): container finished" podID="a3ee7c8e-f519-4574-a3be-f10d70926485" containerID="70ceada97cb370cd59ac50d3ebf826c8f79c5b98646f0ac0eace60ba02c5cdc1" exitCode=0
Oct 29 22:14:47.865335 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.864891    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"
Oct 29 22:14:47.865335 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.864932    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m" event={"ID":"a3ee7c8e-f519-4574-a3be-f10d70926485","Type":"ContainerDied","Data":"70ceada97cb370cd59ac50d3ebf826c8f79c5b98646f0ac0eace60ba02c5cdc1"}
Oct 29 22:14:47.865335 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.864987    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m" event={"ID":"a3ee7c8e-f519-4574-a3be-f10d70926485","Type":"ContainerDied","Data":"7bc6dc7646d805dd7d8db4d5c7b7ec72a29518f553a3440450522702b54f2953"}
Oct 29 22:14:47.865335 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.865015    4962 scope.go:122] "RemoveContainer" containerID="70ceada97cb370cd59ac50d3ebf826c8f79c5b98646f0ac0eace60ba02c5cdc1"
Oct 29 22:14:47.867751 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.867731    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"
Oct 29 22:14:47.867886 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.867730    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2" event={"ID":"03b2ca39-a1c7-45bf-8458-f23215d334b1","Type":"ContainerDied","Data":"0bdddababf5c33e304ffd2a7742fac211ad1d27ed4e96f9a4ab36daeea703f14"}
Oct 29 22:14:47.869323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.869275    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"
Oct 29 22:14:47.869412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.869306    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr" event={"ID":"10c5ec48-f48a-4703-a82c-9a6995506125","Type":"ContainerDied","Data":"261f778cd543b1e36e42073cd85349d02847a1a1a02f32b04aa2dfdbd6ec7f85"}
Oct 29 22:14:47.870813 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.870788    4962 generic.go:356] "Generic (PLEG): container finished" podID="71174a8c-e386-4670-942e-3bc750987080" containerID="0c1daee793a4f3895e641aab38ed7e988ab4dc2a0530cccf9d3942a518bd960e" exitCode=0
Oct 29 22:14:47.870935 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.870818    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"
Oct 29 22:14:47.870935 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.870874    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2" event={"ID":"71174a8c-e386-4670-942e-3bc750987080","Type":"ContainerDied","Data":"0c1daee793a4f3895e641aab38ed7e988ab4dc2a0530cccf9d3942a518bd960e"}
Oct 29 22:14:47.870935 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.870925    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2" event={"ID":"71174a8c-e386-4670-942e-3bc750987080","Type":"ContainerDied","Data":"b8f0649fadcbfee3386c89a4f15bdf65ba2aee1f4532002785c048ddcb663838"}
Oct 29 22:14:47.872531 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.872480    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6" event={"ID":"64a76b6c-1542-4fb1-9730-7b770c452b56","Type":"ContainerDied","Data":"c03d6a48ec0b199a94d95c42c0d8845213a8036fda1eef4939a0b36e111d3a35"}
Oct 29 22:14:47.873275 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.873257    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"
Oct 29 22:14:47.878582 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.878565    4962 scope.go:122] "RemoveContainer" containerID="70ceada97cb370cd59ac50d3ebf826c8f79c5b98646f0ac0eace60ba02c5cdc1"
Oct 29 22:14:47.878898 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:14:47.878876    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70ceada97cb370cd59ac50d3ebf826c8f79c5b98646f0ac0eace60ba02c5cdc1\": container with ID starting with 70ceada97cb370cd59ac50d3ebf826c8f79c5b98646f0ac0eace60ba02c5cdc1 not found: ID does not exist" containerID="70ceada97cb370cd59ac50d3ebf826c8f79c5b98646f0ac0eace60ba02c5cdc1"
Oct 29 22:14:47.879010 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.878904    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ceada97cb370cd59ac50d3ebf826c8f79c5b98646f0ac0eace60ba02c5cdc1"} err="failed to get container status \"70ceada97cb370cd59ac50d3ebf826c8f79c5b98646f0ac0eace60ba02c5cdc1\": rpc error: code = NotFound desc = could not find container \"70ceada97cb370cd59ac50d3ebf826c8f79c5b98646f0ac0eace60ba02c5cdc1\": container with ID starting with 70ceada97cb370cd59ac50d3ebf826c8f79c5b98646f0ac0eace60ba02c5cdc1 not found: ID does not exist"
Oct 29 22:14:47.879010 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.878921    4962 scope.go:122] "RemoveContainer" containerID="d6455e4a3c4bf2a3ecc64d4e53198a939f18bdd777da43a81925856a013ca70e"
Oct 29 22:14:47.887803 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.887786    4962 scope.go:122] "RemoveContainer" containerID="c668a05f6f3d9dbc6215843071b014aba97a0397eae32e6077fb52f4bda89af1"
Oct 29 22:14:47.890278 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.890257    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.890412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.890281    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71174a8c-e386-4670-942e-3bc750987080-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.890412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.890329    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.890412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.890342    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp8ht\" (UniqueName: \"kubernetes.io/projected/71174a8c-e386-4670-942e-3bc750987080-kube-api-access-sp8ht\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.890412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.890357    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.890412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.890370    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71174a8c-e386-4670-942e-3bc750987080-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.890412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.890382    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71174a8c-e386-4670-942e-3bc750987080-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.890412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.890396    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3ee7c8e-f519-4574-a3be-f10d70926485-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.890412 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.890409    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3ee7c8e-f519-4574-a3be-f10d70926485-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.890735 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.890422    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71174a8c-e386-4670-942e-3bc750987080-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.890735 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.890436    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzgfb\" (UniqueName: \"kubernetes.io/projected/a3ee7c8e-f519-4574-a3be-f10d70926485-kube-api-access-rzgfb\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.890735 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.890452    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3ee7c8e-f519-4574-a3be-f10d70926485-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:47.896576 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.896559    4962 scope.go:122] "RemoveContainer" containerID="0c1daee793a4f3895e641aab38ed7e988ab4dc2a0530cccf9d3942a518bd960e"
Oct 29 22:14:47.903424 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.903404    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"]
Oct 29 22:14:47.905832 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.905812    4962 scope.go:122] "RemoveContainer" containerID="0c1daee793a4f3895e641aab38ed7e988ab4dc2a0530cccf9d3942a518bd960e"
Oct 29 22:14:47.906135 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:14:47.906108    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c1daee793a4f3895e641aab38ed7e988ab4dc2a0530cccf9d3942a518bd960e\": container with ID starting with 0c1daee793a4f3895e641aab38ed7e988ab4dc2a0530cccf9d3942a518bd960e not found: ID does not exist" containerID="0c1daee793a4f3895e641aab38ed7e988ab4dc2a0530cccf9d3942a518bd960e"
Oct 29 22:14:47.906194 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.906145    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c1daee793a4f3895e641aab38ed7e988ab4dc2a0530cccf9d3942a518bd960e"} err="failed to get container status \"0c1daee793a4f3895e641aab38ed7e988ab4dc2a0530cccf9d3942a518bd960e\": rpc error: code = NotFound desc = could not find container \"0c1daee793a4f3895e641aab38ed7e988ab4dc2a0530cccf9d3942a518bd960e\": container with ID starting with 0c1daee793a4f3895e641aab38ed7e988ab4dc2a0530cccf9d3942a518bd960e not found: ID does not exist"
Oct 29 22:14:47.906194 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.906168    4962 scope.go:122] "RemoveContainer" containerID="fc31966e8412d3b25aade1df656c1c99dff20fed6545634e537352070118b61c"
Oct 29 22:14:47.950730 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.950662    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"]
Oct 29 22:14:47.953021 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.953000    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-vxpl2"]
Oct 29 22:14:47.985095 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.985072    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"]
Oct 29 22:14:47.989136 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:47.989114    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-jbxcr"]
Oct 29 22:14:48.021831 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:48.021800    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"]
Oct 29 22:14:48.025354 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:48.025335    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-743a5e-openshift-default-cd5b4c9d5-fmgg6"]
Oct 29 22:14:48.074039 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:48.074010    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"]
Oct 29 22:14:48.077488 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:48.077462    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-f2r5m"]
Oct 29 22:14:48.110603 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:48.110574    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"]
Oct 29 22:14:48.114347 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:48.114323    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-75db5b-openshift-default-84786bddf5-pgrk2"]
Oct 29 22:14:49.057603 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:49.057571    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03b2ca39-a1c7-45bf-8458-f23215d334b1" path="/var/lib/kubelet/pods/03b2ca39-a1c7-45bf-8458-f23215d334b1/volumes"
Oct 29 22:14:49.058062 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:49.058046    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10c5ec48-f48a-4703-a82c-9a6995506125" path="/var/lib/kubelet/pods/10c5ec48-f48a-4703-a82c-9a6995506125/volumes"
Oct 29 22:14:49.058543 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:49.058529    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="295e3d96-ff42-4be3-b4de-cd85ca4d8578" path="/var/lib/kubelet/pods/295e3d96-ff42-4be3-b4de-cd85ca4d8578/volumes"
Oct 29 22:14:49.059250 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:49.059235    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64a76b6c-1542-4fb1-9730-7b770c452b56" path="/var/lib/kubelet/pods/64a76b6c-1542-4fb1-9730-7b770c452b56/volumes"
Oct 29 22:14:49.059677 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:49.059664    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71174a8c-e386-4670-942e-3bc750987080" path="/var/lib/kubelet/pods/71174a8c-e386-4670-942e-3bc750987080/volumes"
Oct 29 22:14:49.060090 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:49.060076    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ee7c8e-f519-4574-a3be-f10d70926485" path="/var/lib/kubelet/pods/a3ee7c8e-f519-4574-a3be-f10d70926485/volumes"
Oct 29 22:14:49.667093 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:49.667057    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d"]
Oct 29 22:14:49.667346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:49.667298    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d" podUID="caf2ebea-429f-41f3-9234-27a2df85ddf7" containerName="maas-api" containerID="cri-o://986105593c74aa4a9fd28589e110130cbf8a961d32220b53133c220d96a7fe77" gracePeriod=30
Oct 29 22:14:49.810244 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:49.810207    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp"]
Oct 29 22:14:49.810446 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:49.810419    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp" podUID="e08900e4-1f62-471e-b4ca-c5ee2df27d1b" containerName="payload-processing" containerID="cri-o://58bb02b6dc61c4de0bbcd389431bfe229b2ba69d17be127bfdf731a3e4fc36a9" gracePeriod=30
Oct 29 22:14:50.145307 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:50.144495    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp"
Oct 29 22:14:50.243430 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:50.243123    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e08900e4-1f62-471e-b4ca-c5ee2df27d1b-kube-api-access-pvrlr\" (UniqueName: \"kubernetes.io/projected/e08900e4-1f62-471e-b4ca-c5ee2df27d1b-kube-api-access-pvrlr\") pod \"e08900e4-1f62-471e-b4ca-c5ee2df27d1b\" (UID: \"e08900e4-1f62-471e-b4ca-c5ee2df27d1b\") "
Oct 29 22:14:50.243878 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:50.243842    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e08900e4-1f62-471e-b4ca-c5ee2df27d1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e08900e4-1f62-471e-b4ca-c5ee2df27d1b-plugins-config-volume\") pod \"e08900e4-1f62-471e-b4ca-c5ee2df27d1b\" (UID: \"e08900e4-1f62-471e-b4ca-c5ee2df27d1b\") "
Oct 29 22:14:50.244891 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:50.244866    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e08900e4-1f62-471e-b4ca-c5ee2df27d1b-plugins-config-volume" pod "e08900e4-1f62-471e-b4ca-c5ee2df27d1b" (UID: "e08900e4-1f62-471e-b4ca-c5ee2df27d1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:50.251261 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:50.251219    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e08900e4-1f62-471e-b4ca-c5ee2df27d1b-kube-api-access-pvrlr" pod "e08900e4-1f62-471e-b4ca-c5ee2df27d1b" (UID: "e08900e4-1f62-471e-b4ca-c5ee2df27d1b"). InnerVolumeSpecName "kube-api-access-pvrlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:50.345938 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:50.345904    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e08900e4-1f62-471e-b4ca-c5ee2df27d1b-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:50.345938 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:50.345936    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvrlr\" (UniqueName: \"kubernetes.io/projected/e08900e4-1f62-471e-b4ca-c5ee2df27d1b-kube-api-access-pvrlr\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:50.899609 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:50.899576    4962 generic.go:356] "Generic (PLEG): container finished" podID="e08900e4-1f62-471e-b4ca-c5ee2df27d1b" containerID="58bb02b6dc61c4de0bbcd389431bfe229b2ba69d17be127bfdf731a3e4fc36a9" exitCode=0
Oct 29 22:14:50.899751 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:50.899625    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp"
Oct 29 22:14:50.899751 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:50.899646    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp" event={"ID":"e08900e4-1f62-471e-b4ca-c5ee2df27d1b","Type":"ContainerDied","Data":"58bb02b6dc61c4de0bbcd389431bfe229b2ba69d17be127bfdf731a3e4fc36a9"}
Oct 29 22:14:50.899751 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:50.899699    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp" event={"ID":"e08900e4-1f62-471e-b4ca-c5ee2df27d1b","Type":"ContainerDied","Data":"48f4952d5447434492fba7475d2fb8dcaa6e437df01ad612ad3b11f69b19cb89"}
Oct 29 22:14:50.899751 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:50.899722    4962 scope.go:122] "RemoveContainer" containerID="58bb02b6dc61c4de0bbcd389431bfe229b2ba69d17be127bfdf731a3e4fc36a9"
Oct 29 22:14:50.901804 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:50.901776    4962 generic.go:356] "Generic (PLEG): container finished" podID="caf2ebea-429f-41f3-9234-27a2df85ddf7" containerID="986105593c74aa4a9fd28589e110130cbf8a961d32220b53133c220d96a7fe77" exitCode=0
Oct 29 22:14:50.901945 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:50.901867    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d" event={"ID":"caf2ebea-429f-41f3-9234-27a2df85ddf7","Type":"ContainerDied","Data":"986105593c74aa4a9fd28589e110130cbf8a961d32220b53133c220d96a7fe77"}
Oct 29 22:14:50.910608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:50.910586    4962 scope.go:122] "RemoveContainer" containerID="58bb02b6dc61c4de0bbcd389431bfe229b2ba69d17be127bfdf731a3e4fc36a9"
Oct 29 22:14:50.910886 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:14:50.910867    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58bb02b6dc61c4de0bbcd389431bfe229b2ba69d17be127bfdf731a3e4fc36a9\": container with ID starting with 58bb02b6dc61c4de0bbcd389431bfe229b2ba69d17be127bfdf731a3e4fc36a9 not found: ID does not exist" containerID="58bb02b6dc61c4de0bbcd389431bfe229b2ba69d17be127bfdf731a3e4fc36a9"
Oct 29 22:14:50.910962 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:50.910897    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58bb02b6dc61c4de0bbcd389431bfe229b2ba69d17be127bfdf731a3e4fc36a9"} err="failed to get container status \"58bb02b6dc61c4de0bbcd389431bfe229b2ba69d17be127bfdf731a3e4fc36a9\": rpc error: code = NotFound desc = could not find container \"58bb02b6dc61c4de0bbcd389431bfe229b2ba69d17be127bfdf731a3e4fc36a9\": container with ID starting with 58bb02b6dc61c4de0bbcd389431bfe229b2ba69d17be127bfdf731a3e4fc36a9 not found: ID does not exist"
Oct 29 22:14:50.943001 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:50.942979    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d"
Oct 29 22:14:50.949748 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:50.949725    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp"]
Oct 29 22:14:50.961257 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:50.961236    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f363a3-747b47546-v8pdp"]
Oct 29 22:14:51.054912 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:51.054841    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caf2ebea-429f-41f3-9234-27a2df85ddf7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf2ebea-429f-41f3-9234-27a2df85ddf7-metrics-certs\") pod \"caf2ebea-429f-41f3-9234-27a2df85ddf7\" (UID: \"caf2ebea-429f-41f3-9234-27a2df85ddf7\") "
Oct 29 22:14:51.055070 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:51.054915    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caf2ebea-429f-41f3-9234-27a2df85ddf7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caf2ebea-429f-41f3-9234-27a2df85ddf7-maas-api-tls\") pod \"caf2ebea-429f-41f3-9234-27a2df85ddf7\" (UID: \"caf2ebea-429f-41f3-9234-27a2df85ddf7\") "
Oct 29 22:14:51.055070 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:51.054955    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf2ebea-429f-41f3-9234-27a2df85ddf7-kube-api-access-vdmzk\" (UniqueName: \"kubernetes.io/projected/caf2ebea-429f-41f3-9234-27a2df85ddf7-kube-api-access-vdmzk\") pod \"caf2ebea-429f-41f3-9234-27a2df85ddf7\" (UID: \"caf2ebea-429f-41f3-9234-27a2df85ddf7\") "
Oct 29 22:14:51.055198 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:51.055103    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caf2ebea-429f-41f3-9234-27a2df85ddf7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf2ebea-429f-41f3-9234-27a2df85ddf7-openshift-service-ca\") pod \"caf2ebea-429f-41f3-9234-27a2df85ddf7\" (UID: \"caf2ebea-429f-41f3-9234-27a2df85ddf7\") "
Oct 29 22:14:51.055669 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:51.055639    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf2ebea-429f-41f3-9234-27a2df85ddf7-openshift-service-ca" pod "caf2ebea-429f-41f3-9234-27a2df85ddf7" (UID: "caf2ebea-429f-41f3-9234-27a2df85ddf7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:51.057174 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:51.057146    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf2ebea-429f-41f3-9234-27a2df85ddf7-kube-api-access-vdmzk" pod "caf2ebea-429f-41f3-9234-27a2df85ddf7" (UID: "caf2ebea-429f-41f3-9234-27a2df85ddf7"). InnerVolumeSpecName "kube-api-access-vdmzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:51.057174 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:51.057162    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caf2ebea-429f-41f3-9234-27a2df85ddf7-maas-api-tls" pod "caf2ebea-429f-41f3-9234-27a2df85ddf7" (UID: "caf2ebea-429f-41f3-9234-27a2df85ddf7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:51.057371 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:51.057253    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caf2ebea-429f-41f3-9234-27a2df85ddf7-metrics-certs" pod "caf2ebea-429f-41f3-9234-27a2df85ddf7" (UID: "caf2ebea-429f-41f3-9234-27a2df85ddf7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:51.060747 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:51.060722    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e08900e4-1f62-471e-b4ca-c5ee2df27d1b" path="/var/lib/kubelet/pods/e08900e4-1f62-471e-b4ca-c5ee2df27d1b/volumes"
Oct 29 22:14:51.157375 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:51.157342    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caf2ebea-429f-41f3-9234-27a2df85ddf7-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:51.157375 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:51.157365    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caf2ebea-429f-41f3-9234-27a2df85ddf7-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:51.157375 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:51.157374    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdmzk\" (UniqueName: \"kubernetes.io/projected/caf2ebea-429f-41f3-9234-27a2df85ddf7-kube-api-access-vdmzk\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:51.157795 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:51.157384    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf2ebea-429f-41f3-9234-27a2df85ddf7-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:51.911868 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:51.911829    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d" event={"ID":"caf2ebea-429f-41f3-9234-27a2df85ddf7","Type":"ContainerDied","Data":"8983afb15d532efb839896659ae10a840d8aa304e532cf836fa55e1350d1ed17"}
Oct 29 22:14:51.911868 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:51.911855    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d"
Oct 29 22:14:51.912127 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:51.911906    4962 scope.go:122] "RemoveContainer" containerID="986105593c74aa4a9fd28589e110130cbf8a961d32220b53133c220d96a7fe77"
Oct 29 22:14:51.961139 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:51.961113    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d"]
Oct 29 22:14:51.972736 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:51.972710    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f363a3-5d9d995d76-4jz7d"]
Oct 29 22:14:53.057765 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:53.057731    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caf2ebea-429f-41f3-9234-27a2df85ddf7" path="/var/lib/kubelet/pods/caf2ebea-429f-41f3-9234-27a2df85ddf7/volumes"
Oct 29 22:14:53.582901 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:53.582863    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76"]
Oct 29 22:14:53.615211 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:53.615179    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks"]
Oct 29 22:14:55.652991 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.652964    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e74b984d-q7m8g"]
Oct 29 22:14:55.653522 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653505    4962 cpu_manager.go:414] "containerMap: removing container" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main"
Oct 29 22:14:55.653572 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653525    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main"
Oct 29 22:14:55.653572 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653540    4962 cpu_manager.go:414] "containerMap: removing container" podUID="093f784b-0baf-4f14-984e-ec8003db30fb" containerName="istio-proxy"
Oct 29 22:14:55.653572 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653547    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093f784b-0baf-4f14-984e-ec8003db30fb" containerName="istio-proxy"
Oct 29 22:14:55.653572 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653557    4962 cpu_manager.go:414] "containerMap: removing container" podUID="295e3d96-ff42-4be3-b4de-cd85ca4d8578" containerName="istio-proxy"
Oct 29 22:14:55.653572 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653562    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="295e3d96-ff42-4be3-b4de-cd85ca4d8578" containerName="istio-proxy"
Oct 29 22:14:55.653572 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653571    4962 cpu_manager.go:414] "containerMap: removing container" podUID="b72eb074-2855-4dd2-bf0b-e4a42e92d8bb" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653578    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b72eb074-2855-4dd2-bf0b-e4a42e92d8bb" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653589    4962 cpu_manager.go:414] "containerMap: removing container" podUID="2a71a069-94f9-4736-8b9f-d43292554254" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653594    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a71a069-94f9-4736-8b9f-d43292554254" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653604    4962 cpu_manager.go:414] "containerMap: removing container" podUID="f84b8805-aa71-45c6-b053-526ac3787a08" containerName="maas-api"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653608    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f84b8805-aa71-45c6-b053-526ac3787a08" containerName="maas-api"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653616    4962 cpu_manager.go:414] "containerMap: removing container" podUID="a3ee7c8e-f519-4574-a3be-f10d70926485" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653621    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ee7c8e-f519-4574-a3be-f10d70926485" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653628    4962 cpu_manager.go:414] "containerMap: removing container" podUID="590ad9cd-3237-4783-8dea-82e67274b580" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653632    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="590ad9cd-3237-4783-8dea-82e67274b580" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653638    4962 cpu_manager.go:414] "containerMap: removing container" podUID="64a76b6c-1542-4fb1-9730-7b770c452b56" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653642    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a76b6c-1542-4fb1-9730-7b770c452b56" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653657    4962 cpu_manager.go:414] "containerMap: removing container" podUID="10c5ec48-f48a-4703-a82c-9a6995506125" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653664    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10c5ec48-f48a-4703-a82c-9a6995506125" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653673    4962 cpu_manager.go:414] "containerMap: removing container" podUID="a437f4e2-dc10-41de-851a-ee488b08c6f4" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653679    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a437f4e2-dc10-41de-851a-ee488b08c6f4" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653689    4962 cpu_manager.go:414] "containerMap: removing container" podUID="71174a8c-e386-4670-942e-3bc750987080" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653696    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71174a8c-e386-4670-942e-3bc750987080" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653707    4962 cpu_manager.go:414] "containerMap: removing container" podUID="16540bea-446b-46d9-9cde-4c9c34785ce8" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653713    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16540bea-446b-46d9-9cde-4c9c34785ce8" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653723    4962 cpu_manager.go:414] "containerMap: removing container" podUID="caf2ebea-429f-41f3-9234-27a2df85ddf7" containerName="maas-api"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653729    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caf2ebea-429f-41f3-9234-27a2df85ddf7" containerName="maas-api"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653740    4962 cpu_manager.go:414] "containerMap: removing container" podUID="03b2ca39-a1c7-45bf-8458-f23215d334b1" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653748    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03b2ca39-a1c7-45bf-8458-f23215d334b1" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653758    4962 cpu_manager.go:414] "containerMap: removing container" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653764    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653774    4962 cpu_manager.go:414] "containerMap: removing container" podUID="f18a68f6-5a07-41c2-acbe-58a2c98ebdd7" containerName="maas-api"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653781    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f18a68f6-5a07-41c2-acbe-58a2c98ebdd7" containerName="maas-api"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653801    4962 cpu_manager.go:414] "containerMap: removing container" podUID="fae18ff3-cbb6-4399-92e5-80145ffac570" containerName="payload-pre-processing"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653809    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae18ff3-cbb6-4399-92e5-80145ffac570" containerName="payload-pre-processing"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653821    4962 cpu_manager.go:414] "containerMap: removing container" podUID="baea6f61-3440-48ed-868f-7e2bbf77df97" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653827    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baea6f61-3440-48ed-868f-7e2bbf77df97" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653838    4962 cpu_manager.go:414] "containerMap: removing container" podUID="e08900e4-1f62-471e-b4ca-c5ee2df27d1b" containerName="payload-processing"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653846    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e08900e4-1f62-471e-b4ca-c5ee2df27d1b" containerName="payload-processing"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653857    4962 cpu_manager.go:414] "containerMap: removing container" podUID="f06b758b-d09f-4882-a198-d88e91c7f390" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653862    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06b758b-d09f-4882-a198-d88e91c7f390" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653869    4962 cpu_manager.go:414] "containerMap: removing container" podUID="ad28e0d7-1aaa-44b8-a284-d3383d08e5cb" containerName="maas-api"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653873    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad28e0d7-1aaa-44b8-a284-d3383d08e5cb" containerName="maas-api"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653883    4962 cpu_manager.go:414] "containerMap: removing container" podUID="75cf3e4f-0488-4210-ab72-17b7afdeda6a" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653887    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75cf3e4f-0488-4210-ab72-17b7afdeda6a" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653893    4962 cpu_manager.go:414] "containerMap: removing container" podUID="d0b72137-f812-4900-adce-9b88820a6473" containerName="payload-processing"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653898    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0b72137-f812-4900-adce-9b88820a6473" containerName="payload-processing"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653908    4962 cpu_manager.go:414] "containerMap: removing container" podUID="49d31a1c-a9bf-413b-baf8-0ce2a310de72" containerName="maas-api"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653912    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49d31a1c-a9bf-413b-baf8-0ce2a310de72" containerName="maas-api"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653919    4962 cpu_manager.go:414] "containerMap: removing container" podUID="75cf3e4f-0488-4210-ab72-17b7afdeda6a" containerName="istio-proxy"
Oct 29 22:14:55.653923 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.653924    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75cf3e4f-0488-4210-ab72-17b7afdeda6a" containerName="istio-proxy"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654029    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae18ff3-cbb6-4399-92e5-80145ffac570" containerName="payload-pre-processing"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654040    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="03b2ca39-a1c7-45bf-8458-f23215d334b1" containerName="istio-proxy"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654046    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="baea6f61-3440-48ed-868f-7e2bbf77df97" containerName="istio-proxy"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654053    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="e08900e4-1f62-471e-b4ca-c5ee2df27d1b" containerName="payload-processing"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654058    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06b758b-d09f-4882-a198-d88e91c7f390" containerName="istio-proxy"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654063    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad28e0d7-1aaa-44b8-a284-d3383d08e5cb" containerName="maas-api"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654068    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="75cf3e4f-0488-4210-ab72-17b7afdeda6a" containerName="istio-proxy"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654073    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0b72137-f812-4900-adce-9b88820a6473" containerName="payload-processing"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654079    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="093f784b-0baf-4f14-984e-ec8003db30fb" containerName="istio-proxy"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654085    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="49d31a1c-a9bf-413b-baf8-0ce2a310de72" containerName="maas-api"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654091    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="75cf3e4f-0488-4210-ab72-17b7afdeda6a" containerName="istio-proxy"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654097    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="262b3d82-eee9-4a36-b7e3-244aa1641c07" containerName="main"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654103    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="093f784b-0baf-4f14-984e-ec8003db30fb" containerName="istio-proxy"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654108    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="295e3d96-ff42-4be3-b4de-cd85ca4d8578" containerName="istio-proxy"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654113    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="b72eb074-2855-4dd2-bf0b-e4a42e92d8bb" containerName="istio-proxy"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654118    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a71a069-94f9-4736-8b9f-d43292554254" containerName="istio-proxy"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654125    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="f84b8805-aa71-45c6-b053-526ac3787a08" containerName="maas-api"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654130    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ee7c8e-f519-4574-a3be-f10d70926485" containerName="istio-proxy"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654135    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="590ad9cd-3237-4783-8dea-82e67274b580" containerName="istio-proxy"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654140    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a76b6c-1542-4fb1-9730-7b770c452b56" containerName="istio-proxy"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654148    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="10c5ec48-f48a-4703-a82c-9a6995506125" containerName="istio-proxy"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654153    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="a437f4e2-dc10-41de-851a-ee488b08c6f4" containerName="istio-proxy"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654158    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="71174a8c-e386-4670-942e-3bc750987080" containerName="istio-proxy"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654163    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="caf2ebea-429f-41f3-9234-27a2df85ddf7" containerName="maas-api"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654168    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="16540bea-446b-46d9-9cde-4c9c34785ce8" containerName="istio-proxy"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654172    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce61403b-7eab-4ba2-94e0-982fcea9e829" containerName="main"
Oct 29 22:14:55.655558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.654177    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="f18a68f6-5a07-41c2-acbe-58a2c98ebdd7" containerName="maas-api"
Oct 29 22:14:55.659000 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.658979    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74b984d-q7m8g"
Oct 29 22:14:55.681106 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.681082    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e74b984d-q7m8g"]
Oct 29 22:14:55.759832 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.759809    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:14:55.759983 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.759927    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:14:55.819273 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.819242    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aebb8907-436e-460b-a5a5-9fc278a2ada4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e74b984d-q7m8g\" (UID: \"aebb8907-436e-460b-a5a5-9fc278a2ada4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74b984d-q7m8g"
Oct 29 22:14:55.819400 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.819363    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f54wj\" (UniqueName: \"kubernetes.io/projected/aebb8907-436e-460b-a5a5-9fc278a2ada4-kube-api-access-f54wj\") pod \"maas-api-revoke-sub-e74b984d-q7m8g\" (UID: \"aebb8907-436e-460b-a5a5-9fc278a2ada4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74b984d-q7m8g"
Oct 29 22:14:55.819446 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.819435    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aebb8907-436e-460b-a5a5-9fc278a2ada4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e74b984d-q7m8g\" (UID: \"aebb8907-436e-460b-a5a5-9fc278a2ada4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74b984d-q7m8g"
Oct 29 22:14:55.897158 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.897134    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"]
Oct 29 22:14:55.920744 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.920719    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aebb8907-436e-460b-a5a5-9fc278a2ada4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e74b984d-q7m8g\" (UID: \"aebb8907-436e-460b-a5a5-9fc278a2ada4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74b984d-q7m8g"
Oct 29 22:14:55.920893 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.920880    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aebb8907-436e-460b-a5a5-9fc278a2ada4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e74b984d-q7m8g\" (UID: \"aebb8907-436e-460b-a5a5-9fc278a2ada4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74b984d-q7m8g"
Oct 29 22:14:55.920935 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.920919    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f54wj\" (UniqueName: \"kubernetes.io/projected/aebb8907-436e-460b-a5a5-9fc278a2ada4-kube-api-access-f54wj\") pod \"maas-api-revoke-sub-e74b984d-q7m8g\" (UID: \"aebb8907-436e-460b-a5a5-9fc278a2ada4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74b984d-q7m8g"
Oct 29 22:14:55.921337 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.921315    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aebb8907-436e-460b-a5a5-9fc278a2ada4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e74b984d-q7m8g\" (UID: \"aebb8907-436e-460b-a5a5-9fc278a2ada4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74b984d-q7m8g"
Oct 29 22:14:55.922998 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.922976    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aebb8907-436e-460b-a5a5-9fc278a2ada4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e74b984d-q7m8g\" (UID: \"aebb8907-436e-460b-a5a5-9fc278a2ada4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74b984d-q7m8g"
Oct 29 22:14:55.945841 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.945814    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f54wj\" (UniqueName: \"kubernetes.io/projected/aebb8907-436e-460b-a5a5-9fc278a2ada4-kube-api-access-f54wj\") pod \"maas-api-revoke-sub-e74b984d-q7m8g\" (UID: \"aebb8907-436e-460b-a5a5-9fc278a2ada4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74b984d-q7m8g"
Oct 29 22:14:55.975266 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:55.975246    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74b984d-q7m8g"
Oct 29 22:14:56.277678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:56.277652    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e74b984d-q7m8g"]
Oct 29 22:14:56.279350 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:14:56.279321    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaebb8907_436e_460b_a5a5_9fc278a2ada4.slice/crio-94847744375e8cbb0240124351c18fe3ce66113c93a3c3c2f2b650a501252ebb WatchSource:0}: Error finding container 94847744375e8cbb0240124351c18fe3ce66113c93a3c3c2f2b650a501252ebb: Status 404 returned error can't find the container with id 94847744375e8cbb0240124351c18fe3ce66113c93a3c3c2f2b650a501252ebb
Oct 29 22:14:57.001131 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:57.001094    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74b984d-q7m8g" event={"ID":"aebb8907-436e-460b-a5a5-9fc278a2ada4","Type":"ContainerStarted","Data":"2cbc2910b2465ecdf68375b492eb478f1b62625cf04ce1d13f3053596029580d"}
Oct 29 22:14:57.001531 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:57.001149    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74b984d-q7m8g" event={"ID":"aebb8907-436e-460b-a5a5-9fc278a2ada4","Type":"ContainerStarted","Data":"94847744375e8cbb0240124351c18fe3ce66113c93a3c3c2f2b650a501252ebb"}
Oct 29 22:14:57.046296 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:57.046231    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74b984d-q7m8g" podStartSLOduration=2.04621802 podStartE2EDuration="2.04621802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:14:57.044561727 +0000 UTC m=+1718.117518692" watchObservedRunningTime="2026-10-29 22:14:57.04621802 +0000 UTC m=+1718.119174983"
Oct 29 22:14:58.009175 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:58.009133    4962 generic.go:356] "Generic (PLEG): container finished" podID="aebb8907-436e-460b-a5a5-9fc278a2ada4" containerID="2cbc2910b2465ecdf68375b492eb478f1b62625cf04ce1d13f3053596029580d" exitCode=0
Oct 29 22:14:58.009569 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:58.009218    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74b984d-q7m8g" event={"ID":"aebb8907-436e-460b-a5a5-9fc278a2ada4","Type":"ContainerDied","Data":"2cbc2910b2465ecdf68375b492eb478f1b62625cf04ce1d13f3053596029580d"}
Oct 29 22:14:58.584003 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:58.583964    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76" podUID="fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d" containerName="payload-processing" containerID="cri-o://12848ab4a37ecb12940a56ef55f814d03db31fd79418c0eef3ff0bc5f94373cd" gracePeriod=25
Oct 29 22:14:58.615756 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:58.615726    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks" podUID="ba71124e-36fa-41ea-bb8b-4f4375a1ac7f" containerName="payload-pre-processing" containerID="cri-o://cfc9f9c09b2451d63e6b2b1c7c4651805da87a0ca9f8e8c94246f3f9ea1545d4" gracePeriod=25
Oct 29 22:14:59.170750 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:59.170727    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74b984d-q7m8g"
Oct 29 22:14:59.361279 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:59.361194    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aebb8907-436e-460b-a5a5-9fc278a2ada4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aebb8907-436e-460b-a5a5-9fc278a2ada4-maas-api-openshift-ca\") pod \"aebb8907-436e-460b-a5a5-9fc278a2ada4\" (UID: \"aebb8907-436e-460b-a5a5-9fc278a2ada4\") "
Oct 29 22:14:59.361279 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:59.361248    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aebb8907-436e-460b-a5a5-9fc278a2ada4-kube-api-access-f54wj\" (UniqueName: \"kubernetes.io/projected/aebb8907-436e-460b-a5a5-9fc278a2ada4-kube-api-access-f54wj\") pod \"aebb8907-436e-460b-a5a5-9fc278a2ada4\" (UID: \"aebb8907-436e-460b-a5a5-9fc278a2ada4\") "
Oct 29 22:14:59.361279 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:59.361277    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aebb8907-436e-460b-a5a5-9fc278a2ada4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aebb8907-436e-460b-a5a5-9fc278a2ada4-maas-api-xks-ca\") pod \"aebb8907-436e-460b-a5a5-9fc278a2ada4\" (UID: \"aebb8907-436e-460b-a5a5-9fc278a2ada4\") "
Oct 29 22:14:59.361626 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:59.361604    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aebb8907-436e-460b-a5a5-9fc278a2ada4-maas-api-openshift-ca" pod "aebb8907-436e-460b-a5a5-9fc278a2ada4" (UID: "aebb8907-436e-460b-a5a5-9fc278a2ada4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:14:59.361791 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:59.361770    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aebb8907-436e-460b-a5a5-9fc278a2ada4-maas-api-openshift-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:59.363447 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:59.363422    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aebb8907-436e-460b-a5a5-9fc278a2ada4-maas-api-xks-ca" pod "aebb8907-436e-460b-a5a5-9fc278a2ada4" (UID: "aebb8907-436e-460b-a5a5-9fc278a2ada4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:14:59.363557 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:59.363463    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aebb8907-436e-460b-a5a5-9fc278a2ada4-kube-api-access-f54wj" pod "aebb8907-436e-460b-a5a5-9fc278a2ada4" (UID: "aebb8907-436e-460b-a5a5-9fc278a2ada4"). InnerVolumeSpecName "kube-api-access-f54wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:14:59.462922 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:59.462894    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f54wj\" (UniqueName: \"kubernetes.io/projected/aebb8907-436e-460b-a5a5-9fc278a2ada4-kube-api-access-f54wj\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:14:59.462922 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:14:59.462920    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aebb8907-436e-460b-a5a5-9fc278a2ada4-maas-api-xks-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:00.023697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.023662    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74b984d-q7m8g" event={"ID":"aebb8907-436e-460b-a5a5-9fc278a2ada4","Type":"ContainerDied","Data":"94847744375e8cbb0240124351c18fe3ce66113c93a3c3c2f2b650a501252ebb"}
Oct 29 22:15:00.023844 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.023719    4962 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94847744375e8cbb0240124351c18fe3ce66113c93a3c3c2f2b650a501252ebb"
Oct 29 22:15:00.023844 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.023673    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e74b984d-q7m8g"
Oct 29 22:15:00.149494 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.149460    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr"]
Oct 29 22:15:00.192331 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.171434    4962 cpu_manager.go:414] "containerMap: removing container" podUID="093f784b-0baf-4f14-984e-ec8003db30fb" containerName="istio-proxy"
Oct 29 22:15:00.197086 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.197054    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093f784b-0baf-4f14-984e-ec8003db30fb" containerName="istio-proxy"
Oct 29 22:15:00.197421 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.197402    4962 cpu_manager.go:414] "containerMap: removing container" podUID="aebb8907-436e-460b-a5a5-9fc278a2ada4" containerName="revoke-keys"
Oct 29 22:15:00.197572 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.197558    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aebb8907-436e-460b-a5a5-9fc278a2ada4" containerName="revoke-keys"
Oct 29 22:15:00.197713 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.197701    4962 cpu_manager.go:414] "containerMap: removing container" podUID="03b2ca39-a1c7-45bf-8458-f23215d334b1" containerName="istio-proxy"
Oct 29 22:15:00.197821 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.197809    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03b2ca39-a1c7-45bf-8458-f23215d334b1" containerName="istio-proxy"
Oct 29 22:15:00.203538 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.203515    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="aebb8907-436e-460b-a5a5-9fc278a2ada4" containerName="revoke-keys"
Oct 29 22:15:00.203722 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.203708    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="03b2ca39-a1c7-45bf-8458-f23215d334b1" containerName="istio-proxy"
Oct 29 22:15:00.278960 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.278514    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb"]
Oct 29 22:15:00.311314 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.278758    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr"
Oct 29 22:15:00.321585 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.298798    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr"]
Oct 29 22:15:00.321585 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.298918    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb"
Oct 29 22:15:00.328344 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.326555    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb"]
Oct 29 22:15:00.334005 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.333977    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:15:00.387227 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.387180    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a234504-3170-4ea9-aa0a-2d81ce2171a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888535-bl5dr\" (UID: \"4a234504-3170-4ea9-aa0a-2d81ce2171a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr"
Oct 29 22:15:00.387445 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.387331    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bk5v\" (UniqueName: \"kubernetes.io/projected/99bc070b-5d76-4cf5-b485-619e07383f53-kube-api-access-4bk5v\") pod \"maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb\" (UID: \"99bc070b-5d76-4cf5-b485-619e07383f53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb"
Oct 29 22:15:00.387445 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.387381    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99bc070b-5d76-4cf5-b485-619e07383f53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb\" (UID: \"99bc070b-5d76-4cf5-b485-619e07383f53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb"
Oct 29 22:15:00.387536 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.387456    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44pts\" (UniqueName: \"kubernetes.io/projected/4a234504-3170-4ea9-aa0a-2d81ce2171a0-kube-api-access-44pts\") pod \"maas-api-key-cleanup-29888535-bl5dr\" (UID: \"4a234504-3170-4ea9-aa0a-2d81ce2171a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr"
Oct 29 22:15:00.488445 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.488409    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a234504-3170-4ea9-aa0a-2d81ce2171a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888535-bl5dr\" (UID: \"4a234504-3170-4ea9-aa0a-2d81ce2171a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr"
Oct 29 22:15:00.488586 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.488475    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bk5v\" (UniqueName: \"kubernetes.io/projected/99bc070b-5d76-4cf5-b485-619e07383f53-kube-api-access-4bk5v\") pod \"maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb\" (UID: \"99bc070b-5d76-4cf5-b485-619e07383f53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb"
Oct 29 22:15:00.488586 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.488500    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99bc070b-5d76-4cf5-b485-619e07383f53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb\" (UID: \"99bc070b-5d76-4cf5-b485-619e07383f53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb"
Oct 29 22:15:00.488586 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.488536    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44pts\" (UniqueName: \"kubernetes.io/projected/4a234504-3170-4ea9-aa0a-2d81ce2171a0-kube-api-access-44pts\") pod \"maas-api-key-cleanup-29888535-bl5dr\" (UID: \"4a234504-3170-4ea9-aa0a-2d81ce2171a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr"
Oct 29 22:15:00.489103 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.489083    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a234504-3170-4ea9-aa0a-2d81ce2171a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888535-bl5dr\" (UID: \"4a234504-3170-4ea9-aa0a-2d81ce2171a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr"
Oct 29 22:15:00.489143 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.489098    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99bc070b-5d76-4cf5-b485-619e07383f53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb\" (UID: \"99bc070b-5d76-4cf5-b485-619e07383f53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb"
Oct 29 22:15:00.511253 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.511220    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44pts\" (UniqueName: \"kubernetes.io/projected/4a234504-3170-4ea9-aa0a-2d81ce2171a0-kube-api-access-44pts\") pod \"maas-api-key-cleanup-29888535-bl5dr\" (UID: \"4a234504-3170-4ea9-aa0a-2d81ce2171a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr"
Oct 29 22:15:00.511419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.511341    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bk5v\" (UniqueName: \"kubernetes.io/projected/99bc070b-5d76-4cf5-b485-619e07383f53-kube-api-access-4bk5v\") pod \"maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb\" (UID: \"99bc070b-5d76-4cf5-b485-619e07383f53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb"
Oct 29 22:15:00.643597 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.643510    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr"
Oct 29 22:15:00.668844 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:00.668718    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb"
Oct 29 22:15:01.014732 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:01.014708    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb"]
Oct 29 22:15:01.016279 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:15:01.016248    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99bc070b_5d76_4cf5_b485_619e07383f53.slice/crio-1c0cd6d078efab8f0b83c77c92b35f1eb5ab86db12f31b9e377cae152781b0a2 WatchSource:0}: Error finding container 1c0cd6d078efab8f0b83c77c92b35f1eb5ab86db12f31b9e377cae152781b0a2: Status 404 returned error can't find the container with id 1c0cd6d078efab8f0b83c77c92b35f1eb5ab86db12f31b9e377cae152781b0a2
Oct 29 22:15:01.027194 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:01.027166    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr"]
Oct 29 22:15:01.029482 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:15:01.029449    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a234504_3170_4ea9_aa0a_2d81ce2171a0.slice/crio-82aebecb7594124f48b4f734f8c375917e81682b56f22b53625d41b279715674 WatchSource:0}: Error finding container 82aebecb7594124f48b4f734f8c375917e81682b56f22b53625d41b279715674: Status 404 returned error can't find the container with id 82aebecb7594124f48b4f734f8c375917e81682b56f22b53625d41b279715674
Oct 29 22:15:01.032040 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:01.032015    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb" event={"ID":"99bc070b-5d76-4cf5-b485-619e07383f53","Type":"ContainerStarted","Data":"1c0cd6d078efab8f0b83c77c92b35f1eb5ab86db12f31b9e377cae152781b0a2"}
Oct 29 22:15:02.045158 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:02.045119    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb" event={"ID":"99bc070b-5d76-4cf5-b485-619e07383f53","Type":"ContainerStarted","Data":"4065cc7cf44ce7386c81f349ca0de7f02d9e4c192f6f3615b777b1ab10ccc608"}
Oct 29 22:15:02.048849 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:02.048810    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr" event={"ID":"4a234504-3170-4ea9-aa0a-2d81ce2171a0","Type":"ContainerStarted","Data":"d165e0dd58b42106ac3af4413d729dda399756bfc8b1f8e7b7fb112f5ddd1a31"}
Oct 29 22:15:02.049855 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:02.049832    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr" event={"ID":"4a234504-3170-4ea9-aa0a-2d81ce2171a0","Type":"ContainerStarted","Data":"82aebecb7594124f48b4f734f8c375917e81682b56f22b53625d41b279715674"}
Oct 29 22:15:02.157518 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:02.155491    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr" podStartSLOduration=2.155470471 podStartE2EDuration="2.155470471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:02.154761947 +0000 UTC m=+1723.227718922" watchObservedRunningTime="2026-10-29 22:15:02.155470471 +0000 UTC m=+1723.228427436"
Oct 29 22:15:02.159355 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:02.158556    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb" podStartSLOduration=2.158540679 podStartE2EDuration="2.158540679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:15:02.100699955 +0000 UTC m=+1723.173656920" watchObservedRunningTime="2026-10-29 22:15:02.158540679 +0000 UTC m=+1723.231497645"
Oct 29 22:15:02.940131 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:02.938742    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" containerID="cri-o://7bdb41fc2ff92c2ea0ac3dc8e850a794cd4a87550bc77654fac522041e223711" gracePeriod=45
Oct 29 22:15:03.138312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:03.131558    4962 generic.go:356] "Generic (PLEG): container finished" podID="99bc070b-5d76-4cf5-b485-619e07383f53" containerID="4065cc7cf44ce7386c81f349ca0de7f02d9e4c192f6f3615b777b1ab10ccc608" exitCode=60
Oct 29 22:15:03.138312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:03.131700    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb" event={"ID":"99bc070b-5d76-4cf5-b485-619e07383f53","Type":"ContainerDied","Data":"4065cc7cf44ce7386c81f349ca0de7f02d9e4c192f6f3615b777b1ab10ccc608"}
Oct 29 22:15:03.138312 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:03.132102    4962 scope.go:122] "RemoveContainer" containerID="4065cc7cf44ce7386c81f349ca0de7f02d9e4c192f6f3615b777b1ab10ccc608"
Oct 29 22:15:03.219468 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:03.219421    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": dial tcp 10.132.0.104:8000: connect: connection refused"
Oct 29 22:15:03.255652 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:03.241592    4962 generic.go:356] "Generic (PLEG): container finished" podID="4a234504-3170-4ea9-aa0a-2d81ce2171a0" containerID="d165e0dd58b42106ac3af4413d729dda399756bfc8b1f8e7b7fb112f5ddd1a31" exitCode=60
Oct 29 22:15:03.255652 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:03.242024    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr" event={"ID":"4a234504-3170-4ea9-aa0a-2d81ce2171a0","Type":"ContainerDied","Data":"d165e0dd58b42106ac3af4413d729dda399756bfc8b1f8e7b7fb112f5ddd1a31"}
Oct 29 22:15:03.255652 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:03.243220    4962 scope.go:122] "RemoveContainer" containerID="d165e0dd58b42106ac3af4413d729dda399756bfc8b1f8e7b7fb112f5ddd1a31"
Oct 29 22:15:04.220536 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.213353    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:15:04.220536 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.213964    4962 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:15:04.228304 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.224567    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8000/ready\": dial tcp 10.132.0.104:8000: connect: connection refused"
Oct 29 22:15:04.228304 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.224662    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:15:04.348770 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.286002    4962 patch_prober.go:29] interesting pod/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:15:04.348770 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.286075    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt" podUID="730edf47-8b07-477a-acfc-47faaaacfc84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:15:04.348770 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.326748    4962 generic.go:356] "Generic (PLEG): container finished" podID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerID="7bdb41fc2ff92c2ea0ac3dc8e850a794cd4a87550bc77654fac522041e223711" exitCode=0
Oct 29 22:15:04.348770 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.326903    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" event={"ID":"1e7bb25d-994d-4600-8ade-890853b6f0d3","Type":"ContainerDied","Data":"7bdb41fc2ff92c2ea0ac3dc8e850a794cd4a87550bc77654fac522041e223711"}
Oct 29 22:15:04.348770 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.330210    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb" event={"ID":"99bc070b-5d76-4cf5-b485-619e07383f53","Type":"ContainerStarted","Data":"a1bb6eb61d615d876b690283a586b38e3edbea5c42add34bf667df0a03b24097"}
Oct 29 22:15:04.671220 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.671180    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76" podUID="fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:15:04.737608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.737577    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:15:04.786219 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.786184    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-dshm\") pod \"1e7bb25d-994d-4600-8ade-890853b6f0d3\" (UID: \"1e7bb25d-994d-4600-8ade-890853b6f0d3\") "
Oct 29 22:15:04.786867 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.786844    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e7bb25d-994d-4600-8ade-890853b6f0d3-kube-api-access-ksd44\" (UniqueName: \"kubernetes.io/projected/1e7bb25d-994d-4600-8ade-890853b6f0d3-kube-api-access-ksd44\") pod \"1e7bb25d-994d-4600-8ade-890853b6f0d3\" (UID: \"1e7bb25d-994d-4600-8ade-890853b6f0d3\") "
Oct 29 22:15:04.787366 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.787347    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e7bb25d-994d-4600-8ade-890853b6f0d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e7bb25d-994d-4600-8ade-890853b6f0d3-tls-certs\") pod \"1e7bb25d-994d-4600-8ade-890853b6f0d3\" (UID: \"1e7bb25d-994d-4600-8ade-890853b6f0d3\") "
Oct 29 22:15:04.787569 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.787551    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-model-cache\") pod \"1e7bb25d-994d-4600-8ade-890853b6f0d3\" (UID: \"1e7bb25d-994d-4600-8ade-890853b6f0d3\") "
Oct 29 22:15:04.787773 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.787759    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-home\" (UniqueName: \"kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-home\") pod \"1e7bb25d-994d-4600-8ade-890853b6f0d3\" (UID: \"1e7bb25d-994d-4600-8ade-890853b6f0d3\") "
Oct 29 22:15:04.787882 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.787870    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-tmp-dir\") pod \"1e7bb25d-994d-4600-8ade-890853b6f0d3\" (UID: \"1e7bb25d-994d-4600-8ade-890853b6f0d3\") "
Oct 29 22:15:04.788944 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.788914    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-tmp-dir" pod "1e7bb25d-994d-4600-8ade-890853b6f0d3" (UID: "1e7bb25d-994d-4600-8ade-890853b6f0d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:04.789153 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.789131    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-home" pod "1e7bb25d-994d-4600-8ade-890853b6f0d3" (UID: "1e7bb25d-994d-4600-8ade-890853b6f0d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:04.789452 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.789427    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-model-cache" pod "1e7bb25d-994d-4600-8ade-890853b6f0d3" (UID: "1e7bb25d-994d-4600-8ade-890853b6f0d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:04.789766 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.789745    4962 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-model-cache\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:04.789847 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.789786    4962 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-home\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:04.789847 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.789802    4962 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-tmp-dir\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:04.790764 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.790729    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e7bb25d-994d-4600-8ade-890853b6f0d3-kube-api-access-ksd44" pod "1e7bb25d-994d-4600-8ade-890853b6f0d3" (UID: "1e7bb25d-994d-4600-8ade-890853b6f0d3"). InnerVolumeSpecName "kube-api-access-ksd44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:04.791403 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.791364    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-dshm" pod "1e7bb25d-994d-4600-8ade-890853b6f0d3" (UID: "1e7bb25d-994d-4600-8ade-890853b6f0d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:04.799588 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.798659    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e7bb25d-994d-4600-8ade-890853b6f0d3-tls-certs" pod "1e7bb25d-994d-4600-8ade-890853b6f0d3" (UID: "1e7bb25d-994d-4600-8ade-890853b6f0d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:04.896100 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.895965    4962 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e7bb25d-994d-4600-8ade-890853b6f0d3-dshm\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:04.896100 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.896004    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksd44\" (UniqueName: \"kubernetes.io/projected/1e7bb25d-994d-4600-8ade-890853b6f0d3-kube-api-access-ksd44\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:04.896100 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:04.896023    4962 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e7bb25d-994d-4600-8ade-890853b6f0d3-tls-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:05.342554 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:05.342525    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"
Oct 29 22:15:05.343015 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:05.342558    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j" event={"ID":"1e7bb25d-994d-4600-8ade-890853b6f0d3","Type":"ContainerDied","Data":"cbc22d21fdd53d80e227fb3635cf5f94eda9aea7dabce1eadc310c4e0ea46c8a"}
Oct 29 22:15:05.343015 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:05.342607    4962 scope.go:122] "RemoveContainer" containerID="7bdb41fc2ff92c2ea0ac3dc8e850a794cd4a87550bc77654fac522041e223711"
Oct 29 22:15:05.344609 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:05.344585    4962 generic.go:356] "Generic (PLEG): container finished" podID="99bc070b-5d76-4cf5-b485-619e07383f53" containerID="a1bb6eb61d615d876b690283a586b38e3edbea5c42add34bf667df0a03b24097" exitCode=60
Oct 29 22:15:05.344962 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:05.344943    4962 scope.go:122] "RemoveContainer" containerID="a1bb6eb61d615d876b690283a586b38e3edbea5c42add34bf667df0a03b24097"
Oct 29 22:15:05.345104 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:05.344696    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb" event={"ID":"99bc070b-5d76-4cf5-b485-619e07383f53","Type":"ContainerDied","Data":"a1bb6eb61d615d876b690283a586b38e3edbea5c42add34bf667df0a03b24097"}
Oct 29 22:15:05.345239 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:15:05.345214    4962 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb_odh-ai-gateway-infra(99bc070b-5d76-4cf5-b485-619e07383f53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb" podUID="99bc070b-5d76-4cf5-b485-619e07383f53"
Oct 29 22:15:05.346832 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:05.346812    4962 generic.go:356] "Generic (PLEG): container finished" podID="4a234504-3170-4ea9-aa0a-2d81ce2171a0" containerID="bec6c242d2c0761fe4fc78cea59a0ed15d3fc422b1dbac31fc8980a6958083fa" exitCode=60
Oct 29 22:15:05.346934 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:05.346850    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr" event={"ID":"4a234504-3170-4ea9-aa0a-2d81ce2171a0","Type":"ContainerDied","Data":"bec6c242d2c0761fe4fc78cea59a0ed15d3fc422b1dbac31fc8980a6958083fa"}
Oct 29 22:15:05.347151 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:05.347133    4962 scope.go:122] "RemoveContainer" containerID="bec6c242d2c0761fe4fc78cea59a0ed15d3fc422b1dbac31fc8980a6958083fa"
Oct 29 22:15:05.347430 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:15:05.347410    4962 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888535-bl5dr_odh-ai-gateway-infra(4a234504-3170-4ea9-aa0a-2d81ce2171a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr" podUID="4a234504-3170-4ea9-aa0a-2d81ce2171a0"
Oct 29 22:15:05.353595 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:05.353574    4962 scope.go:122] "RemoveContainer" containerID="4065cc7cf44ce7386c81f349ca0de7f02d9e4c192f6f3615b777b1ab10ccc608"
Oct 29 22:15:05.366138 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:05.366123    4962 scope.go:122] "RemoveContainer" containerID="d165e0dd58b42106ac3af4413d729dda399756bfc8b1f8e7b7fb112f5ddd1a31"
Oct 29 22:15:05.392857 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:05.392834    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"]
Oct 29 22:15:05.399176 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:05.399158    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-e4e24e/distinct-2-w0-e4e24e-kserve-794bfdc995-crt9j"]
Oct 29 22:15:05.607515 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:05.607451    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks" podUID="ba71124e-36fa-41ea-bb8b-4f4375a1ac7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:15:06.357187 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:06.357157    4962 scope.go:122] "RemoveContainer" containerID="a1bb6eb61d615d876b690283a586b38e3edbea5c42add34bf667df0a03b24097"
Oct 29 22:15:06.357868 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:15:06.357441    4962 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb_odh-ai-gateway-infra(99bc070b-5d76-4cf5-b485-619e07383f53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb" podUID="99bc070b-5d76-4cf5-b485-619e07383f53"
Oct 29 22:15:06.358927 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:06.358905    4962 scope.go:122] "RemoveContainer" containerID="bec6c242d2c0761fe4fc78cea59a0ed15d3fc422b1dbac31fc8980a6958083fa"
Oct 29 22:15:06.359115 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:15:06.359095    4962 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888535-bl5dr_odh-ai-gateway-infra(4a234504-3170-4ea9-aa0a-2d81ce2171a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr" podUID="4a234504-3170-4ea9-aa0a-2d81ce2171a0"
Oct 29 22:15:07.057781 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:07.057749    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" path="/var/lib/kubelet/pods/1e7bb25d-994d-4600-8ade-890853b6f0d3/volumes"
Oct 29 22:15:09.720494 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:15:09.720460    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:15:10.917638 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:10.917601    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" containerID="cri-o://f09e58bbf1adfe9b64f3fd8d2e1b9fe2b8c5b9ca695f8e59103da252b7b18fee" gracePeriod=45
Oct 29 22:15:11.188752 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.188730    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:15:11.289007 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.288975    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ac13470-5074-4ded-8f00-e98621700e54-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ac13470-5074-4ded-8f00-e98621700e54-tls-certs\") pod \"5ac13470-5074-4ded-8f00-e98621700e54\" (UID: \"5ac13470-5074-4ded-8f00-e98621700e54\") "
Oct 29 22:15:11.289194 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.289071    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-model-cache\") pod \"5ac13470-5074-4ded-8f00-e98621700e54\" (UID: \"5ac13470-5074-4ded-8f00-e98621700e54\") "
Oct 29 22:15:11.289194 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.289111    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ac13470-5074-4ded-8f00-e98621700e54-kube-api-access-9l267\" (UniqueName: \"kubernetes.io/projected/5ac13470-5074-4ded-8f00-e98621700e54-kube-api-access-9l267\") pod \"5ac13470-5074-4ded-8f00-e98621700e54\" (UID: \"5ac13470-5074-4ded-8f00-e98621700e54\") "
Oct 29 22:15:11.289194 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.289158    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-tmp-dir\") pod \"5ac13470-5074-4ded-8f00-e98621700e54\" (UID: \"5ac13470-5074-4ded-8f00-e98621700e54\") "
Oct 29 22:15:11.289391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.289236    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-dshm\") pod \"5ac13470-5074-4ded-8f00-e98621700e54\" (UID: \"5ac13470-5074-4ded-8f00-e98621700e54\") "
Oct 29 22:15:11.289391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.289333    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-home\" (UniqueName: \"kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-home\") pod \"5ac13470-5074-4ded-8f00-e98621700e54\" (UID: \"5ac13470-5074-4ded-8f00-e98621700e54\") "
Oct 29 22:15:11.289391 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.289348    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-model-cache" pod "5ac13470-5074-4ded-8f00-e98621700e54" (UID: "5ac13470-5074-4ded-8f00-e98621700e54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:11.289563 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.289513    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-tmp-dir" pod "5ac13470-5074-4ded-8f00-e98621700e54" (UID: "5ac13470-5074-4ded-8f00-e98621700e54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:11.289667 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.289638    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-home" pod "5ac13470-5074-4ded-8f00-e98621700e54" (UID: "5ac13470-5074-4ded-8f00-e98621700e54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:11.289914 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.289896    4962 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-model-cache\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:11.289999 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.289916    4962 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-tmp-dir\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:11.289999 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.289928    4962 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-home\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:11.291543 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.291517    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac13470-5074-4ded-8f00-e98621700e54-kube-api-access-9l267" pod "5ac13470-5074-4ded-8f00-e98621700e54" (UID: "5ac13470-5074-4ded-8f00-e98621700e54"). InnerVolumeSpecName "kube-api-access-9l267". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:11.291651 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.291633    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ac13470-5074-4ded-8f00-e98621700e54-tls-certs" pod "5ac13470-5074-4ded-8f00-e98621700e54" (UID: "5ac13470-5074-4ded-8f00-e98621700e54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:11.291699 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.291666    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-dshm" pod "5ac13470-5074-4ded-8f00-e98621700e54" (UID: "5ac13470-5074-4ded-8f00-e98621700e54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:11.391015 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.390983    4962 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ac13470-5074-4ded-8f00-e98621700e54-tls-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:11.391015 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.391014    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l267\" (UniqueName: \"kubernetes.io/projected/5ac13470-5074-4ded-8f00-e98621700e54-kube-api-access-9l267\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:11.391274 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.391026    4962 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ac13470-5074-4ded-8f00-e98621700e54-dshm\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:11.391784 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.391760    4962 generic.go:356] "Generic (PLEG): container finished" podID="5ac13470-5074-4ded-8f00-e98621700e54" containerID="f09e58bbf1adfe9b64f3fd8d2e1b9fe2b8c5b9ca695f8e59103da252b7b18fee" exitCode=0
Oct 29 22:15:11.391850 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.391806    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"
Oct 29 22:15:11.391888 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.391837    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" event={"ID":"5ac13470-5074-4ded-8f00-e98621700e54","Type":"ContainerDied","Data":"f09e58bbf1adfe9b64f3fd8d2e1b9fe2b8c5b9ca695f8e59103da252b7b18fee"}
Oct 29 22:15:11.391888 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.391882    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" event={"ID":"5ac13470-5074-4ded-8f00-e98621700e54","Type":"ContainerDied","Data":"cb0a03ce7a2cf7b1ab85997ac46d016a55db67500339795ed95aba3299b145f8"}
Oct 29 22:15:11.391957 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.391904    4962 scope.go:122] "RemoveContainer" containerID="f09e58bbf1adfe9b64f3fd8d2e1b9fe2b8c5b9ca695f8e59103da252b7b18fee"
Oct 29 22:15:11.402032 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.402009    4962 scope.go:122] "RemoveContainer" containerID="f09e58bbf1adfe9b64f3fd8d2e1b9fe2b8c5b9ca695f8e59103da252b7b18fee"
Oct 29 22:15:11.402328 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:15:11.402296    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f09e58bbf1adfe9b64f3fd8d2e1b9fe2b8c5b9ca695f8e59103da252b7b18fee\": container with ID starting with f09e58bbf1adfe9b64f3fd8d2e1b9fe2b8c5b9ca695f8e59103da252b7b18fee not found: ID does not exist" containerID="f09e58bbf1adfe9b64f3fd8d2e1b9fe2b8c5b9ca695f8e59103da252b7b18fee"
Oct 29 22:15:11.402423 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.402334    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f09e58bbf1adfe9b64f3fd8d2e1b9fe2b8c5b9ca695f8e59103da252b7b18fee"} err="failed to get container status \"f09e58bbf1adfe9b64f3fd8d2e1b9fe2b8c5b9ca695f8e59103da252b7b18fee\": rpc error: code = NotFound desc = could not find container \"f09e58bbf1adfe9b64f3fd8d2e1b9fe2b8c5b9ca695f8e59103da252b7b18fee\": container with ID starting with f09e58bbf1adfe9b64f3fd8d2e1b9fe2b8c5b9ca695f8e59103da252b7b18fee not found: ID does not exist"
Oct 29 22:15:11.459161 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.459101    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"]
Oct 29 22:15:11.461583 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:11.461562    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67"]
Oct 29 22:15:12.160887 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:12.160843    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-f363a3/ipp-route-f363a3-kserve-5bfbf4ccc-f8s67" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:15:13.057566 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:13.057533    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ac13470-5074-4ded-8f00-e98621700e54" path="/var/lib/kubelet/pods/5ac13470-5074-4ded-8f00-e98621700e54/volumes"
Oct 29 22:15:14.586343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:14.586299    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76" podUID="fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:15:15.606829 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:15.606786    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks" podUID="ba71124e-36fa-41ea-bb8b-4f4375a1ac7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:15:15.919799 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:15.919764    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"]
Oct 29 22:15:15.920101 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:15.920049    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5" podUID="9762d8d0-a759-42ac-bc1b-c948adfa6f9d" containerName="istio-proxy" containerID="cri-o://f766610cabea37b19826f94ee2348ea64754f505943d291f6644290f47b70486" gracePeriod=30
Oct 29 22:15:15.929672 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:15.929578    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"]
Oct 29 22:15:15.929813 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:15.929773    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr" podUID="4ef87fe2-429a-4393-8ddf-fa43e2dd9044" containerName="istio-proxy" containerID="cri-o://7cd80b4e67697abfff4dcd0c68f6f8feae551f87b263441e3db252dfde348611" gracePeriod=30
Oct 29 22:15:15.935304 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:15.935264    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"]
Oct 29 22:15:15.935502 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:15.935476    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw" podUID="f6631eff-5d71-483c-aac7-8d578274ce76" containerName="istio-proxy" containerID="cri-o://31e08276344e9ca4d0dbd2b1b01de955fb414adce60b060b1b11350b0d6b9e7c" gracePeriod=30
Oct 29 22:15:16.055553 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:16.055026    4962 scope.go:122] "RemoveContainer" containerID="a1bb6eb61d615d876b690283a586b38e3edbea5c42add34bf667df0a03b24097"
Oct 29 22:15:17.050603 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:17.050574    4962 scope.go:122] "RemoveContainer" containerID="bec6c242d2c0761fe4fc78cea59a0ed15d3fc422b1dbac31fc8980a6958083fa"
Oct 29 22:15:17.447810 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:17.447781    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr" event={"ID":"4a234504-3170-4ea9-aa0a-2d81ce2171a0","Type":"ContainerStarted","Data":"b76dfacd7c0178976d94e21bcea1ca2019332353e0c9ca32a19e844268d1b5a1"}
Oct 29 22:15:17.449540 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:17.449518    4962 generic.go:356] "Generic (PLEG): container finished" podID="99bc070b-5d76-4cf5-b485-619e07383f53" containerID="b2a2e0bd55ad47905997c44a9bf686a8cad236b12298b4be8a2ccac4c8f06b42" exitCode=60
Oct 29 22:15:17.449650 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:17.449591    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb" event={"ID":"99bc070b-5d76-4cf5-b485-619e07383f53","Type":"ContainerDied","Data":"b2a2e0bd55ad47905997c44a9bf686a8cad236b12298b4be8a2ccac4c8f06b42"}
Oct 29 22:15:17.449696 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:17.449647    4962 scope.go:122] "RemoveContainer" containerID="a1bb6eb61d615d876b690283a586b38e3edbea5c42add34bf667df0a03b24097"
Oct 29 22:15:17.449956 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:17.449939    4962 scope.go:122] "RemoveContainer" containerID="b2a2e0bd55ad47905997c44a9bf686a8cad236b12298b4be8a2ccac4c8f06b42"
Oct 29 22:15:17.450200 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:15:17.450180    4962 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb_odh-ai-gateway-infra(99bc070b-5d76-4cf5-b485-619e07383f53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb" podUID="99bc070b-5d76-4cf5-b485-619e07383f53"
Oct 29 22:15:18.458616 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:18.458583    4962 generic.go:356] "Generic (PLEG): container finished" podID="4a234504-3170-4ea9-aa0a-2d81ce2171a0" containerID="b76dfacd7c0178976d94e21bcea1ca2019332353e0c9ca32a19e844268d1b5a1" exitCode=60
Oct 29 22:15:18.459051 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:18.458653    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr" event={"ID":"4a234504-3170-4ea9-aa0a-2d81ce2171a0","Type":"ContainerDied","Data":"b76dfacd7c0178976d94e21bcea1ca2019332353e0c9ca32a19e844268d1b5a1"}
Oct 29 22:15:18.459051 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:18.458714    4962 scope.go:122] "RemoveContainer" containerID="bec6c242d2c0761fe4fc78cea59a0ed15d3fc422b1dbac31fc8980a6958083fa"
Oct 29 22:15:18.459202 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:18.459052    4962 scope.go:122] "RemoveContainer" containerID="b76dfacd7c0178976d94e21bcea1ca2019332353e0c9ca32a19e844268d1b5a1"
Oct 29 22:15:18.459348 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:15:18.459325    4962 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888535-bl5dr_odh-ai-gateway-infra(4a234504-3170-4ea9-aa0a-2d81ce2171a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr" podUID="4a234504-3170-4ea9-aa0a-2d81ce2171a0"
Oct 29 22:15:18.500086 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:18.500058    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr"]
Oct 29 22:15:18.541071 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:18.541042    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb"]
Oct 29 22:15:18.762313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:18.761900    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb"
Oct 29 22:15:18.810335 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:18.808729    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99bc070b-5d76-4cf5-b485-619e07383f53-kube-api-access-4bk5v\" (UniqueName: \"kubernetes.io/projected/99bc070b-5d76-4cf5-b485-619e07383f53-kube-api-access-4bk5v\") pod \"99bc070b-5d76-4cf5-b485-619e07383f53\" (UID: \"99bc070b-5d76-4cf5-b485-619e07383f53\") "
Oct 29 22:15:18.810335 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:18.808953    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99bc070b-5d76-4cf5-b485-619e07383f53-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99bc070b-5d76-4cf5-b485-619e07383f53-maas-api-service-ca\") pod \"99bc070b-5d76-4cf5-b485-619e07383f53\" (UID: \"99bc070b-5d76-4cf5-b485-619e07383f53\") "
Oct 29 22:15:18.810335 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:18.809708    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99bc070b-5d76-4cf5-b485-619e07383f53-maas-api-service-ca" pod "99bc070b-5d76-4cf5-b485-619e07383f53" (UID: "99bc070b-5d76-4cf5-b485-619e07383f53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:18.815109 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:18.815077    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99bc070b-5d76-4cf5-b485-619e07383f53-kube-api-access-4bk5v" pod "99bc070b-5d76-4cf5-b485-619e07383f53" (UID: "99bc070b-5d76-4cf5-b485-619e07383f53"). InnerVolumeSpecName "kube-api-access-4bk5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:18.877536 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:18.877504    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76"
Oct 29 22:15:18.910461 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:18.910432    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bk5v\" (UniqueName: \"kubernetes.io/projected/99bc070b-5d76-4cf5-b485-619e07383f53-kube-api-access-4bk5v\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:18.910461 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:18.910459    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99bc070b-5d76-4cf5-b485-619e07383f53-maas-api-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:18.941323 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:18.941300    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks"
Oct 29 22:15:19.011643 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.011562    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba71124e-36fa-41ea-bb8b-4f4375a1ac7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba71124e-36fa-41ea-bb8b-4f4375a1ac7f-plugins-config-volume\") pod \"ba71124e-36fa-41ea-bb8b-4f4375a1ac7f\" (UID: \"ba71124e-36fa-41ea-bb8b-4f4375a1ac7f\") "
Oct 29 22:15:19.011799 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.011666    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba71124e-36fa-41ea-bb8b-4f4375a1ac7f-kube-api-access-rhmnq\" (UniqueName: \"kubernetes.io/projected/ba71124e-36fa-41ea-bb8b-4f4375a1ac7f-kube-api-access-rhmnq\") pod \"ba71124e-36fa-41ea-bb8b-4f4375a1ac7f\" (UID: \"ba71124e-36fa-41ea-bb8b-4f4375a1ac7f\") "
Oct 29 22:15:19.011799 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.011736    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d-plugins-config-volume\") pod \"fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d\" (UID: \"fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d\") "
Oct 29 22:15:19.011799 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.011775    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d-kube-api-access-wmlrg\" (UniqueName: \"kubernetes.io/projected/fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d-kube-api-access-wmlrg\") pod \"fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d\" (UID: \"fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d\") "
Oct 29 22:15:19.011983 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.011958    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba71124e-36fa-41ea-bb8b-4f4375a1ac7f-plugins-config-volume" pod "ba71124e-36fa-41ea-bb8b-4f4375a1ac7f" (UID: "ba71124e-36fa-41ea-bb8b-4f4375a1ac7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:19.012156 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.012131    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d-plugins-config-volume" pod "fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d" (UID: "fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:19.012361 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.012341    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba71124e-36fa-41ea-bb8b-4f4375a1ac7f-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:19.012454 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.012365    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:19.013975 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.013947    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba71124e-36fa-41ea-bb8b-4f4375a1ac7f-kube-api-access-rhmnq" pod "ba71124e-36fa-41ea-bb8b-4f4375a1ac7f" (UID: "ba71124e-36fa-41ea-bb8b-4f4375a1ac7f"). InnerVolumeSpecName "kube-api-access-rhmnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:19.014259 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.014238    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d-kube-api-access-wmlrg" pod "fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d" (UID: "fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d"). InnerVolumeSpecName "kube-api-access-wmlrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:19.113837 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.113814    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhmnq\" (UniqueName: \"kubernetes.io/projected/ba71124e-36fa-41ea-bb8b-4f4375a1ac7f-kube-api-access-rhmnq\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:19.113837 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.113838    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmlrg\" (UniqueName: \"kubernetes.io/projected/fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d-kube-api-access-wmlrg\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:19.467728 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.467694    4962 generic.go:356] "Generic (PLEG): container finished" podID="ba71124e-36fa-41ea-bb8b-4f4375a1ac7f" containerID="cfc9f9c09b2451d63e6b2b1c7c4651805da87a0ca9f8e8c94246f3f9ea1545d4" exitCode=0
Oct 29 22:15:19.468179 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.467739    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks"
Oct 29 22:15:19.468179 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.467786    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks" event={"ID":"ba71124e-36fa-41ea-bb8b-4f4375a1ac7f","Type":"ContainerDied","Data":"cfc9f9c09b2451d63e6b2b1c7c4651805da87a0ca9f8e8c94246f3f9ea1545d4"}
Oct 29 22:15:19.468179 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.467840    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks" event={"ID":"ba71124e-36fa-41ea-bb8b-4f4375a1ac7f","Type":"ContainerDied","Data":"13c93476dfb3228bf138acaa627bf51c62090a58dd7639f268b67ab2eb96e733"}
Oct 29 22:15:19.468179 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.467862    4962 scope.go:122] "RemoveContainer" containerID="cfc9f9c09b2451d63e6b2b1c7c4651805da87a0ca9f8e8c94246f3f9ea1545d4"
Oct 29 22:15:19.469210 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.469128    4962 generic.go:356] "Generic (PLEG): container finished" podID="fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d" containerID="12848ab4a37ecb12940a56ef55f814d03db31fd79418c0eef3ff0bc5f94373cd" exitCode=0
Oct 29 22:15:19.469210 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.469184    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76"
Oct 29 22:15:19.469210 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.469198    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76" event={"ID":"fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d","Type":"ContainerDied","Data":"12848ab4a37ecb12940a56ef55f814d03db31fd79418c0eef3ff0bc5f94373cd"}
Oct 29 22:15:19.469441 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.469240    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76" event={"ID":"fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d","Type":"ContainerDied","Data":"e058b8a951a50694a65a9519623693c90cc17bf0a97510461ba52d87d2863cf1"}
Oct 29 22:15:19.470909 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.470886    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb"
Oct 29 22:15:19.471027 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.470943    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb" event={"ID":"99bc070b-5d76-4cf5-b485-619e07383f53","Type":"ContainerDied","Data":"1c0cd6d078efab8f0b83c77c92b35f1eb5ab86db12f31b9e377cae152781b0a2"}
Oct 29 22:15:19.478649 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.478625    4962 scope.go:122] "RemoveContainer" containerID="cfc9f9c09b2451d63e6b2b1c7c4651805da87a0ca9f8e8c94246f3f9ea1545d4"
Oct 29 22:15:19.478936 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:15:19.478911    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfc9f9c09b2451d63e6b2b1c7c4651805da87a0ca9f8e8c94246f3f9ea1545d4\": container with ID starting with cfc9f9c09b2451d63e6b2b1c7c4651805da87a0ca9f8e8c94246f3f9ea1545d4 not found: ID does not exist" containerID="cfc9f9c09b2451d63e6b2b1c7c4651805da87a0ca9f8e8c94246f3f9ea1545d4"
Oct 29 22:15:19.479018 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.478947    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfc9f9c09b2451d63e6b2b1c7c4651805da87a0ca9f8e8c94246f3f9ea1545d4"} err="failed to get container status \"cfc9f9c09b2451d63e6b2b1c7c4651805da87a0ca9f8e8c94246f3f9ea1545d4\": rpc error: code = NotFound desc = could not find container \"cfc9f9c09b2451d63e6b2b1c7c4651805da87a0ca9f8e8c94246f3f9ea1545d4\": container with ID starting with cfc9f9c09b2451d63e6b2b1c7c4651805da87a0ca9f8e8c94246f3f9ea1545d4 not found: ID does not exist"
Oct 29 22:15:19.479018 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.478970    4962 scope.go:122] "RemoveContainer" containerID="12848ab4a37ecb12940a56ef55f814d03db31fd79418c0eef3ff0bc5f94373cd"
Oct 29 22:15:19.488027 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.488005    4962 scope.go:122] "RemoveContainer" containerID="12848ab4a37ecb12940a56ef55f814d03db31fd79418c0eef3ff0bc5f94373cd"
Oct 29 22:15:19.488344 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:15:19.488323    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12848ab4a37ecb12940a56ef55f814d03db31fd79418c0eef3ff0bc5f94373cd\": container with ID starting with 12848ab4a37ecb12940a56ef55f814d03db31fd79418c0eef3ff0bc5f94373cd not found: ID does not exist" containerID="12848ab4a37ecb12940a56ef55f814d03db31fd79418c0eef3ff0bc5f94373cd"
Oct 29 22:15:19.488419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.488351    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12848ab4a37ecb12940a56ef55f814d03db31fd79418c0eef3ff0bc5f94373cd"} err="failed to get container status \"12848ab4a37ecb12940a56ef55f814d03db31fd79418c0eef3ff0bc5f94373cd\": rpc error: code = NotFound desc = could not find container \"12848ab4a37ecb12940a56ef55f814d03db31fd79418c0eef3ff0bc5f94373cd\": container with ID starting with 12848ab4a37ecb12940a56ef55f814d03db31fd79418c0eef3ff0bc5f94373cd not found: ID does not exist"
Oct 29 22:15:19.488419 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.488367    4962 scope.go:122] "RemoveContainer" containerID="b2a2e0bd55ad47905997c44a9bf686a8cad236b12298b4be8a2ccac4c8f06b42"
Oct 29 22:15:19.520318 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.520250    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb"]
Oct 29 22:15:19.530497 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.530468    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-a9e67f-29888535-wz4rb"]
Oct 29 22:15:19.555238 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.555205    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks"]
Oct 29 22:15:19.562102 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.562074    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-e4e24e-7d799d8fb7-557ks"]
Oct 29 22:15:19.596966 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.596925    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76"]
Oct 29 22:15:19.601511 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.601415    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-e4e24e-5cd44579cb-nvx76"]
Oct 29 22:15:19.610665 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.610645    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr"
Oct 29 22:15:19.732353 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.732210    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a234504-3170-4ea9-aa0a-2d81ce2171a0-kube-api-access-44pts\" (UniqueName: \"kubernetes.io/projected/4a234504-3170-4ea9-aa0a-2d81ce2171a0-kube-api-access-44pts\") pod \"4a234504-3170-4ea9-aa0a-2d81ce2171a0\" (UID: \"4a234504-3170-4ea9-aa0a-2d81ce2171a0\") "
Oct 29 22:15:19.732556 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.732365    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a234504-3170-4ea9-aa0a-2d81ce2171a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a234504-3170-4ea9-aa0a-2d81ce2171a0-maas-api-service-ca\") pod \"4a234504-3170-4ea9-aa0a-2d81ce2171a0\" (UID: \"4a234504-3170-4ea9-aa0a-2d81ce2171a0\") "
Oct 29 22:15:19.733470 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.733083    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a234504-3170-4ea9-aa0a-2d81ce2171a0-maas-api-service-ca" pod "4a234504-3170-4ea9-aa0a-2d81ce2171a0" (UID: "4a234504-3170-4ea9-aa0a-2d81ce2171a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:19.741747 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.741715    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a234504-3170-4ea9-aa0a-2d81ce2171a0-kube-api-access-44pts" pod "4a234504-3170-4ea9-aa0a-2d81ce2171a0" (UID: "4a234504-3170-4ea9-aa0a-2d81ce2171a0"). InnerVolumeSpecName "kube-api-access-44pts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:19.833349 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.833321    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44pts\" (UniqueName: \"kubernetes.io/projected/4a234504-3170-4ea9-aa0a-2d81ce2171a0-kube-api-access-44pts\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:19.833349 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:19.833346    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a234504-3170-4ea9-aa0a-2d81ce2171a0-maas-api-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:20.489023 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:20.488978    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr" event={"ID":"4a234504-3170-4ea9-aa0a-2d81ce2171a0","Type":"ContainerDied","Data":"82aebecb7594124f48b4f734f8c375917e81682b56f22b53625d41b279715674"}
Oct 29 22:15:20.489558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:20.489102    4962 scope.go:122] "RemoveContainer" containerID="b76dfacd7c0178976d94e21bcea1ca2019332353e0c9ca32a19e844268d1b5a1"
Oct 29 22:15:20.489558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:20.489269    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr"
Oct 29 22:15:20.538435 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:20.538405    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr"]
Oct 29 22:15:20.541957 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:20.541925    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888535-bl5dr"]
Oct 29 22:15:21.062542 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.060514    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a234504-3170-4ea9-aa0a-2d81ce2171a0" path="/var/lib/kubelet/pods/4a234504-3170-4ea9-aa0a-2d81ce2171a0/volumes"
Oct 29 22:15:21.062542 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.061152    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99bc070b-5d76-4cf5-b485-619e07383f53" path="/var/lib/kubelet/pods/99bc070b-5d76-4cf5-b485-619e07383f53/volumes"
Oct 29 22:15:21.062542 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.061712    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba71124e-36fa-41ea-bb8b-4f4375a1ac7f" path="/var/lib/kubelet/pods/ba71124e-36fa-41ea-bb8b-4f4375a1ac7f/volumes"
Oct 29 22:15:21.062762 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.062628    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d" path="/var/lib/kubelet/pods/fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d/volumes"
Oct 29 22:15:21.214593 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.214523    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 22:15:21.260970 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.260921    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 22:15:21.336043 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.336010    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 22:15:21.363341 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.363308    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-podinfo\") pod \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") "
Oct 29 22:15:21.363500 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.363367    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-workload-socket\") pod \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") "
Oct 29 22:15:21.363500 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.363417    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-envoy\") pod \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") "
Oct 29 22:15:21.363500 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.363474    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-workload-certs\") pod \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") "
Oct 29 22:15:21.363674 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.363512    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-token\" (UniqueName: \"kubernetes.io/projected/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-token\") pod \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") "
Oct 29 22:15:21.363674 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.363588    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-workload-certs\") pod \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") "
Oct 29 22:15:21.363674 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.363650    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-podinfo\") pod \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") "
Oct 29 22:15:21.363811 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.363754    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-workload-certs" pod "4ef87fe2-429a-4393-8ddf-fa43e2dd9044" (UID: "4ef87fe2-429a-4393-8ddf-fa43e2dd9044"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.363911 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.363887    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-workload-certs" pod "9762d8d0-a759-42ac-bc1b-c948adfa6f9d" (UID: "9762d8d0-a759-42ac-bc1b-c948adfa6f9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.363992 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.363910    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-workload-socket" pod "9762d8d0-a759-42ac-bc1b-c948adfa6f9d" (UID: "9762d8d0-a759-42ac-bc1b-c948adfa6f9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.364055 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.363988    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istiod-ca-cert\") pod \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") "
Oct 29 22:15:21.364055 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.364042    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-token\" (UniqueName: \"kubernetes.io/projected/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-token\") pod \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") "
Oct 29 22:15:21.364159 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.364080    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-data\") pod \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") "
Oct 29 22:15:21.364159 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.364137    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-data\") pod \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") "
Oct 29 22:15:21.364267 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.364166    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-kube-api-access-jwksj\" (UniqueName: \"kubernetes.io/projected/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-kube-api-access-jwksj\") pod \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") "
Oct 29 22:15:21.364267 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.364215    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-credential-socket\") pod \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") "
Oct 29 22:15:21.364359 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.364299    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-workload-socket\") pod \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") "
Oct 29 22:15:21.364392 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.364363    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-envoy\") pod \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") "
Oct 29 22:15:21.364432 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.364407    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-credential-socket\") pod \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\" (UID: \"4ef87fe2-429a-4393-8ddf-fa43e2dd9044\") "
Oct 29 22:15:21.365443 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.364516    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istiod-ca-cert" pod "4ef87fe2-429a-4393-8ddf-fa43e2dd9044" (UID: "4ef87fe2-429a-4393-8ddf-fa43e2dd9044"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:21.365443 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.364527    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istiod-ca-cert\") pod \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") "
Oct 29 22:15:21.365443 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.364751    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-kube-api-access-htxvl\" (UniqueName: \"kubernetes.io/projected/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-kube-api-access-htxvl\") pod \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\" (UID: \"9762d8d0-a759-42ac-bc1b-c948adfa6f9d\") "
Oct 29 22:15:21.365443 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.364839    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istiod-ca-cert" pod "9762d8d0-a759-42ac-bc1b-c948adfa6f9d" (UID: "9762d8d0-a759-42ac-bc1b-c948adfa6f9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:21.365443 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.365108    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-credential-socket" pod "9762d8d0-a759-42ac-bc1b-c948adfa6f9d" (UID: "9762d8d0-a759-42ac-bc1b-c948adfa6f9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.365443 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.365164    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-credential-socket" pod "4ef87fe2-429a-4393-8ddf-fa43e2dd9044" (UID: "4ef87fe2-429a-4393-8ddf-fa43e2dd9044"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.365443 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.365182    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-data" pod "9762d8d0-a759-42ac-bc1b-c948adfa6f9d" (UID: "9762d8d0-a759-42ac-bc1b-c948adfa6f9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.365443 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.365411    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.365443 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.365435    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.365443 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.365451    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.366057 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.365465    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.366057 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.365478    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.366057 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.365491    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.366057 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.365495    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-workload-socket" pod "4ef87fe2-429a-4393-8ddf-fa43e2dd9044" (UID: "4ef87fe2-429a-4393-8ddf-fa43e2dd9044"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.366057 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.365504    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.366057 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.365539    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.366057 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.365695    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-data" pod "4ef87fe2-429a-4393-8ddf-fa43e2dd9044" (UID: "4ef87fe2-429a-4393-8ddf-fa43e2dd9044"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.368078 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.368042    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-envoy" pod "9762d8d0-a759-42ac-bc1b-c948adfa6f9d" (UID: "9762d8d0-a759-42ac-bc1b-c948adfa6f9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.368252 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.368228    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-token" pod "9762d8d0-a759-42ac-bc1b-c948adfa6f9d" (UID: "9762d8d0-a759-42ac-bc1b-c948adfa6f9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:21.368478 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.368385    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-kube-api-access-htxvl" pod "9762d8d0-a759-42ac-bc1b-c948adfa6f9d" (UID: "9762d8d0-a759-42ac-bc1b-c948adfa6f9d"). InnerVolumeSpecName "kube-api-access-htxvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:21.368783 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.368748    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-podinfo" pod "4ef87fe2-429a-4393-8ddf-fa43e2dd9044" (UID: "4ef87fe2-429a-4393-8ddf-fa43e2dd9044"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:15:21.369508 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.369441    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-envoy" pod "4ef87fe2-429a-4393-8ddf-fa43e2dd9044" (UID: "4ef87fe2-429a-4393-8ddf-fa43e2dd9044"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.369697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.369575    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-token" pod "4ef87fe2-429a-4393-8ddf-fa43e2dd9044" (UID: "4ef87fe2-429a-4393-8ddf-fa43e2dd9044"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:21.369825 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.369724    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-kube-api-access-jwksj" pod "4ef87fe2-429a-4393-8ddf-fa43e2dd9044" (UID: "4ef87fe2-429a-4393-8ddf-fa43e2dd9044"). InnerVolumeSpecName "kube-api-access-jwksj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:21.370371 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.370347    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-podinfo" pod "9762d8d0-a759-42ac-bc1b-c948adfa6f9d" (UID: "9762d8d0-a759-42ac-bc1b-c948adfa6f9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:15:21.466836 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.466764    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6631eff-5d71-483c-aac7-8d578274ce76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6631eff-5d71-483c-aac7-8d578274ce76-istiod-ca-cert\") pod \"f6631eff-5d71-483c-aac7-8d578274ce76\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") "
Oct 29 22:15:21.466836 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.466816    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-istio-data\") pod \"f6631eff-5d71-483c-aac7-8d578274ce76\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") "
Oct 29 22:15:21.467041 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.466857    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-istio-envoy\") pod \"f6631eff-5d71-483c-aac7-8d578274ce76\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") "
Oct 29 22:15:21.467041 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.466962    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-workload-certs\") pod \"f6631eff-5d71-483c-aac7-8d578274ce76\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") "
Oct 29 22:15:21.467163 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.467078    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6631eff-5d71-483c-aac7-8d578274ce76-kube-api-access-q88b5\" (UniqueName: \"kubernetes.io/projected/f6631eff-5d71-483c-aac7-8d578274ce76-kube-api-access-q88b5\") pod \"f6631eff-5d71-483c-aac7-8d578274ce76\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") "
Oct 29 22:15:21.467163 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.467138    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-credential-socket\") pod \"f6631eff-5d71-483c-aac7-8d578274ce76\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") "
Oct 29 22:15:21.467272 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.467155    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-istio-data" pod "f6631eff-5d71-483c-aac7-8d578274ce76" (UID: "f6631eff-5d71-483c-aac7-8d578274ce76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.467272 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.467214    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-workload-socket\") pod \"f6631eff-5d71-483c-aac7-8d578274ce76\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") "
Oct 29 22:15:21.467272 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.467237    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6631eff-5d71-483c-aac7-8d578274ce76-istiod-ca-cert" pod "f6631eff-5d71-483c-aac7-8d578274ce76" (UID: "f6631eff-5d71-483c-aac7-8d578274ce76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:21.467458 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.467318    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6631eff-5d71-483c-aac7-8d578274ce76-istio-token\" (UniqueName: \"kubernetes.io/projected/f6631eff-5d71-483c-aac7-8d578274ce76-istio-token\") pod \"f6631eff-5d71-483c-aac7-8d578274ce76\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") "
Oct 29 22:15:21.467458 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.467369    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6631eff-5d71-483c-aac7-8d578274ce76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6631eff-5d71-483c-aac7-8d578274ce76-istio-podinfo\") pod \"f6631eff-5d71-483c-aac7-8d578274ce76\" (UID: \"f6631eff-5d71-483c-aac7-8d578274ce76\") "
Oct 29 22:15:21.467458 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.467410    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-credential-socket" pod "f6631eff-5d71-483c-aac7-8d578274ce76" (UID: "f6631eff-5d71-483c-aac7-8d578274ce76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.467786 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.467502    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-workload-certs" pod "f6631eff-5d71-483c-aac7-8d578274ce76" (UID: "f6631eff-5d71-483c-aac7-8d578274ce76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.467786 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.467571    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-workload-socket" pod "f6631eff-5d71-483c-aac7-8d578274ce76" (UID: "f6631eff-5d71-483c-aac7-8d578274ce76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.468004 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.467844    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.468004 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.467863    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6631eff-5d71-483c-aac7-8d578274ce76-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.468004 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.467876    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.468004 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.467888    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.468004 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.467901    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.468004 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.467913    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwksj\" (UniqueName: \"kubernetes.io/projected/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-kube-api-access-jwksj\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.468004 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.467927    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.468004 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.467940    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.468004 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.467952    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ef87fe2-429a-4393-8ddf-fa43e2dd9044-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.468004 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.467967    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.468004 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.467981    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htxvl\" (UniqueName: \"kubernetes.io/projected/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-kube-api-access-htxvl\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.468004 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.467994    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.468004 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.468007    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9762d8d0-a759-42ac-bc1b-c948adfa6f9d-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.469894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.469867    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6631eff-5d71-483c-aac7-8d578274ce76-istio-token" pod "f6631eff-5d71-483c-aac7-8d578274ce76" (UID: "f6631eff-5d71-483c-aac7-8d578274ce76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:21.469977 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.469886    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6631eff-5d71-483c-aac7-8d578274ce76-kube-api-access-q88b5" pod "f6631eff-5d71-483c-aac7-8d578274ce76" (UID: "f6631eff-5d71-483c-aac7-8d578274ce76"). InnerVolumeSpecName "kube-api-access-q88b5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:21.470283 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.470261    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6631eff-5d71-483c-aac7-8d578274ce76-istio-podinfo" pod "f6631eff-5d71-483c-aac7-8d578274ce76" (UID: "f6631eff-5d71-483c-aac7-8d578274ce76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:15:21.470833 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.470812    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-istio-envoy" pod "f6631eff-5d71-483c-aac7-8d578274ce76" (UID: "f6631eff-5d71-483c-aac7-8d578274ce76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:21.503657 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.503628    4962 generic.go:356] "Generic (PLEG): container finished" podID="4ef87fe2-429a-4393-8ddf-fa43e2dd9044" containerID="7cd80b4e67697abfff4dcd0c68f6f8feae551f87b263441e3db252dfde348611" exitCode=0
Oct 29 22:15:21.504093 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.503673    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"
Oct 29 22:15:21.504093 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.503702    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr" event={"ID":"4ef87fe2-429a-4393-8ddf-fa43e2dd9044","Type":"ContainerDied","Data":"7cd80b4e67697abfff4dcd0c68f6f8feae551f87b263441e3db252dfde348611"}
Oct 29 22:15:21.504093 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.503754    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr" event={"ID":"4ef87fe2-429a-4393-8ddf-fa43e2dd9044","Type":"ContainerDied","Data":"dd1eeaf3327a80b0c05e37a819df4750e4ff0631b0fd5bfab28f77c3ef33d6aa"}
Oct 29 22:15:21.504093 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.503777    4962 scope.go:122] "RemoveContainer" containerID="7cd80b4e67697abfff4dcd0c68f6f8feae551f87b263441e3db252dfde348611"
Oct 29 22:15:21.505736 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.505715    4962 generic.go:356] "Generic (PLEG): container finished" podID="f6631eff-5d71-483c-aac7-8d578274ce76" containerID="31e08276344e9ca4d0dbd2b1b01de955fb414adce60b060b1b11350b0d6b9e7c" exitCode=0
Oct 29 22:15:21.505885 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.505783    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw" event={"ID":"f6631eff-5d71-483c-aac7-8d578274ce76","Type":"ContainerDied","Data":"31e08276344e9ca4d0dbd2b1b01de955fb414adce60b060b1b11350b0d6b9e7c"}
Oct 29 22:15:21.505885 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.505831    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw" event={"ID":"f6631eff-5d71-483c-aac7-8d578274ce76","Type":"ContainerDied","Data":"e6184dedc5ce30abdbfd447ace2eb9b1852cd90f3452136f89c20f1171685909"}
Oct 29 22:15:21.505885 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.505790    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"
Oct 29 22:15:21.508180 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.508010    4962 generic.go:356] "Generic (PLEG): container finished" podID="9762d8d0-a759-42ac-bc1b-c948adfa6f9d" containerID="f766610cabea37b19826f94ee2348ea64754f505943d291f6644290f47b70486" exitCode=0
Oct 29 22:15:21.508180 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.508099    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5" event={"ID":"9762d8d0-a759-42ac-bc1b-c948adfa6f9d","Type":"ContainerDied","Data":"f766610cabea37b19826f94ee2348ea64754f505943d291f6644290f47b70486"}
Oct 29 22:15:21.508180 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.508137    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5" event={"ID":"9762d8d0-a759-42ac-bc1b-c948adfa6f9d","Type":"ContainerDied","Data":"f28c29c26c0cf41d80ea682dee47914e761161e5b6844025ee3363d91753dc35"}
Oct 29 22:15:21.514328 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.508065    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"
Oct 29 22:15:21.517477 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.517411    4962 scope.go:122] "RemoveContainer" containerID="7cd80b4e67697abfff4dcd0c68f6f8feae551f87b263441e3db252dfde348611"
Oct 29 22:15:21.517713 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:15:21.517685    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cd80b4e67697abfff4dcd0c68f6f8feae551f87b263441e3db252dfde348611\": container with ID starting with 7cd80b4e67697abfff4dcd0c68f6f8feae551f87b263441e3db252dfde348611 not found: ID does not exist" containerID="7cd80b4e67697abfff4dcd0c68f6f8feae551f87b263441e3db252dfde348611"
Oct 29 22:15:21.517783 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.517725    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cd80b4e67697abfff4dcd0c68f6f8feae551f87b263441e3db252dfde348611"} err="failed to get container status \"7cd80b4e67697abfff4dcd0c68f6f8feae551f87b263441e3db252dfde348611\": rpc error: code = NotFound desc = could not find container \"7cd80b4e67697abfff4dcd0c68f6f8feae551f87b263441e3db252dfde348611\": container with ID starting with 7cd80b4e67697abfff4dcd0c68f6f8feae551f87b263441e3db252dfde348611 not found: ID does not exist"
Oct 29 22:15:21.517854 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.517795    4962 scope.go:122] "RemoveContainer" containerID="31e08276344e9ca4d0dbd2b1b01de955fb414adce60b060b1b11350b0d6b9e7c"
Oct 29 22:15:21.532040 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.531407    4962 scope.go:122] "RemoveContainer" containerID="31e08276344e9ca4d0dbd2b1b01de955fb414adce60b060b1b11350b0d6b9e7c"
Oct 29 22:15:21.532040 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:15:21.531712    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e08276344e9ca4d0dbd2b1b01de955fb414adce60b060b1b11350b0d6b9e7c\": container with ID starting with 31e08276344e9ca4d0dbd2b1b01de955fb414adce60b060b1b11350b0d6b9e7c not found: ID does not exist" containerID="31e08276344e9ca4d0dbd2b1b01de955fb414adce60b060b1b11350b0d6b9e7c"
Oct 29 22:15:21.532040 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.531734    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e08276344e9ca4d0dbd2b1b01de955fb414adce60b060b1b11350b0d6b9e7c"} err="failed to get container status \"31e08276344e9ca4d0dbd2b1b01de955fb414adce60b060b1b11350b0d6b9e7c\": rpc error: code = NotFound desc = could not find container \"31e08276344e9ca4d0dbd2b1b01de955fb414adce60b060b1b11350b0d6b9e7c\": container with ID starting with 31e08276344e9ca4d0dbd2b1b01de955fb414adce60b060b1b11350b0d6b9e7c not found: ID does not exist"
Oct 29 22:15:21.532040 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.531749    4962 scope.go:122] "RemoveContainer" containerID="f766610cabea37b19826f94ee2348ea64754f505943d291f6644290f47b70486"
Oct 29 22:15:21.540940 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.540921    4962 scope.go:122] "RemoveContainer" containerID="f766610cabea37b19826f94ee2348ea64754f505943d291f6644290f47b70486"
Oct 29 22:15:21.541197 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:15:21.541178    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f766610cabea37b19826f94ee2348ea64754f505943d291f6644290f47b70486\": container with ID starting with f766610cabea37b19826f94ee2348ea64754f505943d291f6644290f47b70486 not found: ID does not exist" containerID="f766610cabea37b19826f94ee2348ea64754f505943d291f6644290f47b70486"
Oct 29 22:15:21.541258 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.541204    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f766610cabea37b19826f94ee2348ea64754f505943d291f6644290f47b70486"} err="failed to get container status \"f766610cabea37b19826f94ee2348ea64754f505943d291f6644290f47b70486\": rpc error: code = NotFound desc = could not find container \"f766610cabea37b19826f94ee2348ea64754f505943d291f6644290f47b70486\": container with ID starting with f766610cabea37b19826f94ee2348ea64754f505943d291f6644290f47b70486 not found: ID does not exist"
Oct 29 22:15:21.569756 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.569731    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.569756 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.569754    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.569902 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.569763    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q88b5\" (UniqueName: \"kubernetes.io/projected/f6631eff-5d71-483c-aac7-8d578274ce76-kube-api-access-q88b5\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.569902 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.569774    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6631eff-5d71-483c-aac7-8d578274ce76-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.569902 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.569784    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6631eff-5d71-483c-aac7-8d578274ce76-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.569902 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.569792    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6631eff-5d71-483c-aac7-8d578274ce76-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:21.582063 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.582039    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"]
Oct 29 22:15:21.587959 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.587939    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-hcfg5"]
Oct 29 22:15:21.621835 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.621815    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"]
Oct 29 22:15:21.625366 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.625344    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-dq4tr"]
Oct 29 22:15:21.657616 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.657588    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"]
Oct 29 22:15:21.661122 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:21.661099    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-e4e24e-openshift-default-d477c4d55-4hqzw"]
Oct 29 22:15:23.057233 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:23.057204    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ef87fe2-429a-4393-8ddf-fa43e2dd9044" path="/var/lib/kubelet/pods/4ef87fe2-429a-4393-8ddf-fa43e2dd9044/volumes"
Oct 29 22:15:23.057763 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:23.057746    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9762d8d0-a759-42ac-bc1b-c948adfa6f9d" path="/var/lib/kubelet/pods/9762d8d0-a759-42ac-bc1b-c948adfa6f9d/volumes"
Oct 29 22:15:23.058174 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:23.058161    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6631eff-5d71-483c-aac7-8d578274ce76" path="/var/lib/kubelet/pods/f6631eff-5d71-483c-aac7-8d578274ce76/volumes"
Oct 29 22:15:39.730274 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:15:39.730234    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:15:40.673857 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:40.673819    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"]
Oct 29 22:15:40.674481 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:40.674433    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s" podUID="b4cda533-309f-46e1-8cad-7239630ac19d" containerName="istio-proxy" containerID="cri-o://04c3fdbbfbc59616646d2a8cfc2ead6b536ca689936ba6636e643a935c4478a5" gracePeriod=30
Oct 29 22:15:40.691477 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:40.691450    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"]
Oct 29 22:15:40.691925 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:40.691876    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw" podUID="e8c4fd64-8818-475c-b3d0-579054bcd76a" containerName="istio-proxy" containerID="cri-o://5deab39450f38e650d846ff0ebd3c07ff1e0c246b2c66d6e46784f74bf0818d3" gracePeriod=30
Oct 29 22:15:40.699569 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:40.699542    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"]
Oct 29 22:15:40.700767 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:40.700731    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb" podUID="9da507f9-8669-4b99-99de-8a7a83cd7850" containerName="istio-proxy" containerID="cri-o://818cb4a9e6555cfb76a34b6ac6fb6b2bd9e430b059d01e7e5a0659907bbdc053" gracePeriod=30
Oct 29 22:15:44.292306 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:44.292255    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:15:44.292705 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:44.292326    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw" podUID="e8c4fd64-8818-475c-b3d0-579054bcd76a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:15:44.322901 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:44.322869    4962 patch_prober.go:29] interesting pod/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:15:44.323039 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:44.322917    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s" podUID="b4cda533-309f-46e1-8cad-7239630ac19d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:15:45.940876 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.940849    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:15:45.944999 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.944979    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:15:45.995346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.995264    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4cda533-309f-46e1-8cad-7239630ac19d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cda533-309f-46e1-8cad-7239630ac19d-istio-podinfo\") pod \"b4cda533-309f-46e1-8cad-7239630ac19d\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") "
Oct 29 22:15:45.995491 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.995366    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-podinfo\") pod \"e8c4fd64-8818-475c-b3d0-579054bcd76a\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") "
Oct 29 22:15:45.995491 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.995396    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-data\") pod \"e8c4fd64-8818-475c-b3d0-579054bcd76a\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") "
Oct 29 22:15:45.995491 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.995437    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4cda533-309f-46e1-8cad-7239630ac19d-kube-api-access-nfrhg\" (UniqueName: \"kubernetes.io/projected/b4cda533-309f-46e1-8cad-7239630ac19d-kube-api-access-nfrhg\") pod \"b4cda533-309f-46e1-8cad-7239630ac19d\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") "
Oct 29 22:15:45.995491 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.995469    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-credential-socket\") pod \"b4cda533-309f-46e1-8cad-7239630ac19d\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") "
Oct 29 22:15:45.995734 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.995495    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-istio-envoy\") pod \"b4cda533-309f-46e1-8cad-7239630ac19d\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") "
Oct 29 22:15:45.995734 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.995527    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-istio-data\") pod \"b4cda533-309f-46e1-8cad-7239630ac19d\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") "
Oct 29 22:15:45.995734 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.995563    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-workload-socket\") pod \"b4cda533-309f-46e1-8cad-7239630ac19d\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") "
Oct 29 22:15:45.995734 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.995594    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-workload-certs\") pod \"e8c4fd64-8818-475c-b3d0-579054bcd76a\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") "
Oct 29 22:15:45.995734 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.995656    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-data" pod "e8c4fd64-8818-475c-b3d0-579054bcd76a" (UID: "e8c4fd64-8818-475c-b3d0-579054bcd76a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:45.995734 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.995664    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-envoy\") pod \"e8c4fd64-8818-475c-b3d0-579054bcd76a\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") "
Oct 29 22:15:45.996084 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.995748    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4cda533-309f-46e1-8cad-7239630ac19d-istio-token\" (UniqueName: \"kubernetes.io/projected/b4cda533-309f-46e1-8cad-7239630ac19d-istio-token\") pod \"b4cda533-309f-46e1-8cad-7239630ac19d\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") "
Oct 29 22:15:45.996084 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.995813    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-workload-certs\") pod \"b4cda533-309f-46e1-8cad-7239630ac19d\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") "
Oct 29 22:15:45.996084 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.995852    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8c4fd64-8818-475c-b3d0-579054bcd76a-kube-api-access-6j8bh\" (UniqueName: \"kubernetes.io/projected/e8c4fd64-8818-475c-b3d0-579054bcd76a-kube-api-access-6j8bh\") pod \"e8c4fd64-8818-475c-b3d0-579054bcd76a\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") "
Oct 29 22:15:45.996084 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.995895    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-credential-socket\") pod \"e8c4fd64-8818-475c-b3d0-579054bcd76a\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") "
Oct 29 22:15:45.996084 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.995966    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8c4fd64-8818-475c-b3d0-579054bcd76a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c4fd64-8818-475c-b3d0-579054bcd76a-istiod-ca-cert\") pod \"e8c4fd64-8818-475c-b3d0-579054bcd76a\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") "
Oct 29 22:15:45.996084 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.996012    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-workload-socket" pod "b4cda533-309f-46e1-8cad-7239630ac19d" (UID: "b4cda533-309f-46e1-8cad-7239630ac19d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:45.996084 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.996017    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-workload-socket\") pod \"e8c4fd64-8818-475c-b3d0-579054bcd76a\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") "
Oct 29 22:15:45.996084 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.996052    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-token\" (UniqueName: \"kubernetes.io/projected/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-token\") pod \"e8c4fd64-8818-475c-b3d0-579054bcd76a\" (UID: \"e8c4fd64-8818-475c-b3d0-579054bcd76a\") "
Oct 29 22:15:45.996596 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.996169    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4cda533-309f-46e1-8cad-7239630ac19d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cda533-309f-46e1-8cad-7239630ac19d-istiod-ca-cert\") pod \"b4cda533-309f-46e1-8cad-7239630ac19d\" (UID: \"b4cda533-309f-46e1-8cad-7239630ac19d\") "
Oct 29 22:15:45.996596 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.996200    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-istio-data" pod "b4cda533-309f-46e1-8cad-7239630ac19d" (UID: "b4cda533-309f-46e1-8cad-7239630ac19d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:45.996714 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.996645    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-credential-socket" pod "e8c4fd64-8818-475c-b3d0-579054bcd76a" (UID: "e8c4fd64-8818-475c-b3d0-579054bcd76a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:45.996855 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.996825    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:45.996925 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.996860    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:45.996925 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.996874    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:45.996925 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.996889    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:45.997626 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.997319    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-workload-certs" pod "e8c4fd64-8818-475c-b3d0-579054bcd76a" (UID: "e8c4fd64-8818-475c-b3d0-579054bcd76a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:45.997626 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.997391    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-credential-socket" pod "b4cda533-309f-46e1-8cad-7239630ac19d" (UID: "b4cda533-309f-46e1-8cad-7239630ac19d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:45.997626 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.997426    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8c4fd64-8818-475c-b3d0-579054bcd76a-istiod-ca-cert" pod "e8c4fd64-8818-475c-b3d0-579054bcd76a" (UID: "e8c4fd64-8818-475c-b3d0-579054bcd76a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:45.997626 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.997584    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-workload-certs" pod "b4cda533-309f-46e1-8cad-7239630ac19d" (UID: "b4cda533-309f-46e1-8cad-7239630ac19d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:45.997918 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.997703    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-workload-socket" pod "e8c4fd64-8818-475c-b3d0-579054bcd76a" (UID: "e8c4fd64-8818-475c-b3d0-579054bcd76a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:45.998031 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:45.998010    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4cda533-309f-46e1-8cad-7239630ac19d-istiod-ca-cert" pod "b4cda533-309f-46e1-8cad-7239630ac19d" (UID: "b4cda533-309f-46e1-8cad-7239630ac19d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:46.007587 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.007566    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-podinfo" pod "e8c4fd64-8818-475c-b3d0-579054bcd76a" (UID: "e8c4fd64-8818-475c-b3d0-579054bcd76a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:15:46.009045 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.009007    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-token" pod "e8c4fd64-8818-475c-b3d0-579054bcd76a" (UID: "e8c4fd64-8818-475c-b3d0-579054bcd76a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:46.009365 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.009330    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4cda533-309f-46e1-8cad-7239630ac19d-istio-podinfo" pod "b4cda533-309f-46e1-8cad-7239630ac19d" (UID: "b4cda533-309f-46e1-8cad-7239630ac19d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:15:46.009452 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.009373    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4cda533-309f-46e1-8cad-7239630ac19d-kube-api-access-nfrhg" pod "b4cda533-309f-46e1-8cad-7239630ac19d" (UID: "b4cda533-309f-46e1-8cad-7239630ac19d"). InnerVolumeSpecName "kube-api-access-nfrhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:46.009452 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.009410    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4cda533-309f-46e1-8cad-7239630ac19d-istio-token" pod "b4cda533-309f-46e1-8cad-7239630ac19d" (UID: "b4cda533-309f-46e1-8cad-7239630ac19d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:46.009745 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.009522    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-istio-envoy" pod "b4cda533-309f-46e1-8cad-7239630ac19d" (UID: "b4cda533-309f-46e1-8cad-7239630ac19d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.010040 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.010015    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-envoy" pod "e8c4fd64-8818-475c-b3d0-579054bcd76a" (UID: "e8c4fd64-8818-475c-b3d0-579054bcd76a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.010163 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.010153    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c4fd64-8818-475c-b3d0-579054bcd76a-kube-api-access-6j8bh" pod "e8c4fd64-8818-475c-b3d0-579054bcd76a" (UID: "e8c4fd64-8818-475c-b3d0-579054bcd76a"). InnerVolumeSpecName "kube-api-access-6j8bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:46.034675 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.034654    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:15:46.097443 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.097414    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-podinfo\") pod \"9da507f9-8669-4b99-99de-8a7a83cd7850\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") "
Oct 29 22:15:46.097443 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.097447    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-token\" (UniqueName: \"kubernetes.io/projected/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-token\") pod \"9da507f9-8669-4b99-99de-8a7a83cd7850\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") "
Oct 29 22:15:46.097662 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.097505    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-envoy\") pod \"9da507f9-8669-4b99-99de-8a7a83cd7850\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") "
Oct 29 22:15:46.097662 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.097522    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9da507f9-8669-4b99-99de-8a7a83cd7850-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9da507f9-8669-4b99-99de-8a7a83cd7850-istiod-ca-cert\") pod \"9da507f9-8669-4b99-99de-8a7a83cd7850\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") "
Oct 29 22:15:46.097662 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.097542    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-data\") pod \"9da507f9-8669-4b99-99de-8a7a83cd7850\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") "
Oct 29 22:15:46.097874 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.097850    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-data" pod "9da507f9-8669-4b99-99de-8a7a83cd7850" (UID: "9da507f9-8669-4b99-99de-8a7a83cd7850"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.097966 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.097925    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-credential-socket\") pod \"9da507f9-8669-4b99-99de-8a7a83cd7850\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") "
Oct 29 22:15:46.098098 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098007    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9da507f9-8669-4b99-99de-8a7a83cd7850-kube-api-access-sfk4f\" (UniqueName: \"kubernetes.io/projected/9da507f9-8669-4b99-99de-8a7a83cd7850-kube-api-access-sfk4f\") pod \"9da507f9-8669-4b99-99de-8a7a83cd7850\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") "
Oct 29 22:15:46.098098 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.097940    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9da507f9-8669-4b99-99de-8a7a83cd7850-istiod-ca-cert" pod "9da507f9-8669-4b99-99de-8a7a83cd7850" (UID: "9da507f9-8669-4b99-99de-8a7a83cd7850"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:46.098098 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098073    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-workload-certs\") pod \"9da507f9-8669-4b99-99de-8a7a83cd7850\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") "
Oct 29 22:15:46.098672 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098097    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-credential-socket" pod "9da507f9-8669-4b99-99de-8a7a83cd7850" (UID: "9da507f9-8669-4b99-99de-8a7a83cd7850"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.098672 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098144    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-workload-socket\") pod \"9da507f9-8669-4b99-99de-8a7a83cd7850\" (UID: \"9da507f9-8669-4b99-99de-8a7a83cd7850\") "
Oct 29 22:15:46.098672 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098371    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-workload-certs" pod "9da507f9-8669-4b99-99de-8a7a83cd7850" (UID: "9da507f9-8669-4b99-99de-8a7a83cd7850"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.098672 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098546    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-workload-socket" pod "9da507f9-8669-4b99-99de-8a7a83cd7850" (UID: "9da507f9-8669-4b99-99de-8a7a83cd7850"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.098894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098695    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.098894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098716    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4cda533-309f-46e1-8cad-7239630ac19d-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.098894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098731    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.098894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098745    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j8bh\" (UniqueName: \"kubernetes.io/projected/e8c4fd64-8818-475c-b3d0-579054bcd76a-kube-api-access-6j8bh\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.098894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098761    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c4fd64-8818-475c-b3d0-579054bcd76a-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.098894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098775    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.098894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098788    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.098894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098803    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9da507f9-8669-4b99-99de-8a7a83cd7850-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.098894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098817    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.098894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098830    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.098894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098843    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4cda533-309f-46e1-8cad-7239630ac19d-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.098894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098857    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4cda533-309f-46e1-8cad-7239630ac19d-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.098894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098871    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.098894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098884    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c4fd64-8818-475c-b3d0-579054bcd76a-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.098894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098899    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfrhg\" (UniqueName: \"kubernetes.io/projected/b4cda533-309f-46e1-8cad-7239630ac19d-kube-api-access-nfrhg\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.099708 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098915    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.099708 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098929    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4cda533-309f-46e1-8cad-7239630ac19d-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.099708 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098945    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.099708 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.098961    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c4fd64-8818-475c-b3d0-579054bcd76a-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.100172 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.100152    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da507f9-8669-4b99-99de-8a7a83cd7850-kube-api-access-sfk4f" pod "9da507f9-8669-4b99-99de-8a7a83cd7850" (UID: "9da507f9-8669-4b99-99de-8a7a83cd7850"). InnerVolumeSpecName "kube-api-access-sfk4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:46.100269 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.100248    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-envoy" pod "9da507f9-8669-4b99-99de-8a7a83cd7850" (UID: "9da507f9-8669-4b99-99de-8a7a83cd7850"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:15:46.100386 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.100262    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-token" pod "9da507f9-8669-4b99-99de-8a7a83cd7850" (UID: "9da507f9-8669-4b99-99de-8a7a83cd7850"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:46.100507 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.100468    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-podinfo" pod "9da507f9-8669-4b99-99de-8a7a83cd7850" (UID: "9da507f9-8669-4b99-99de-8a7a83cd7850"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:15:46.200230 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.200205    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.200230 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.200227    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.200393 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.200237    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9da507f9-8669-4b99-99de-8a7a83cd7850-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.200393 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.200245    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfk4f\" (UniqueName: \"kubernetes.io/projected/9da507f9-8669-4b99-99de-8a7a83cd7850-kube-api-access-sfk4f\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:46.722240 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.722210    4962 generic.go:356] "Generic (PLEG): container finished" podID="b4cda533-309f-46e1-8cad-7239630ac19d" containerID="04c3fdbbfbc59616646d2a8cfc2ead6b536ca689936ba6636e643a935c4478a5" exitCode=0
Oct 29 22:15:46.722410 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.722268    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"
Oct 29 22:15:46.722410 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.722313    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s" event={"ID":"b4cda533-309f-46e1-8cad-7239630ac19d","Type":"ContainerDied","Data":"04c3fdbbfbc59616646d2a8cfc2ead6b536ca689936ba6636e643a935c4478a5"}
Oct 29 22:15:46.722410 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.722365    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s" event={"ID":"b4cda533-309f-46e1-8cad-7239630ac19d","Type":"ContainerDied","Data":"d86bfc8e1ad656b6fa7b688132a657e393a4e31b7a444aad34ec8c5d946ad206"}
Oct 29 22:15:46.722410 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.722391    4962 scope.go:122] "RemoveContainer" containerID="04c3fdbbfbc59616646d2a8cfc2ead6b536ca689936ba6636e643a935c4478a5"
Oct 29 22:15:46.723954 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.723933    4962 generic.go:356] "Generic (PLEG): container finished" podID="9da507f9-8669-4b99-99de-8a7a83cd7850" containerID="818cb4a9e6555cfb76a34b6ac6fb6b2bd9e430b059d01e7e5a0659907bbdc053" exitCode=0
Oct 29 22:15:46.724098 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.723975    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"
Oct 29 22:15:46.724098 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.724061    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb" event={"ID":"9da507f9-8669-4b99-99de-8a7a83cd7850","Type":"ContainerDied","Data":"818cb4a9e6555cfb76a34b6ac6fb6b2bd9e430b059d01e7e5a0659907bbdc053"}
Oct 29 22:15:46.724235 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.724098    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb" event={"ID":"9da507f9-8669-4b99-99de-8a7a83cd7850","Type":"ContainerDied","Data":"3df7d5d64c5de0e691ceb32aececaf8b8ca56f4d806a50e228af730b99621238"}
Oct 29 22:15:46.725871 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.725850    4962 generic.go:356] "Generic (PLEG): container finished" podID="e8c4fd64-8818-475c-b3d0-579054bcd76a" containerID="5deab39450f38e650d846ff0ebd3c07ff1e0c246b2c66d6e46784f74bf0818d3" exitCode=0
Oct 29 22:15:46.725960 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.725937    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw" event={"ID":"e8c4fd64-8818-475c-b3d0-579054bcd76a","Type":"ContainerDied","Data":"5deab39450f38e650d846ff0ebd3c07ff1e0c246b2c66d6e46784f74bf0818d3"}
Oct 29 22:15:46.726030 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.725975    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw" event={"ID":"e8c4fd64-8818-475c-b3d0-579054bcd76a","Type":"ContainerDied","Data":"4f352fd661b49576fdc341dc0b51b8669ec6f42f09978d609d3344d78824e3d0"}
Oct 29 22:15:46.726200 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.726039    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"
Oct 29 22:15:46.733754 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.733738    4962 scope.go:122] "RemoveContainer" containerID="04c3fdbbfbc59616646d2a8cfc2ead6b536ca689936ba6636e643a935c4478a5"
Oct 29 22:15:46.734203 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:15:46.734164    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04c3fdbbfbc59616646d2a8cfc2ead6b536ca689936ba6636e643a935c4478a5\": container with ID starting with 04c3fdbbfbc59616646d2a8cfc2ead6b536ca689936ba6636e643a935c4478a5 not found: ID does not exist" containerID="04c3fdbbfbc59616646d2a8cfc2ead6b536ca689936ba6636e643a935c4478a5"
Oct 29 22:15:46.734304 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.734197    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04c3fdbbfbc59616646d2a8cfc2ead6b536ca689936ba6636e643a935c4478a5"} err="failed to get container status \"04c3fdbbfbc59616646d2a8cfc2ead6b536ca689936ba6636e643a935c4478a5\": rpc error: code = NotFound desc = could not find container \"04c3fdbbfbc59616646d2a8cfc2ead6b536ca689936ba6636e643a935c4478a5\": container with ID starting with 04c3fdbbfbc59616646d2a8cfc2ead6b536ca689936ba6636e643a935c4478a5 not found: ID does not exist"
Oct 29 22:15:46.734304 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.734219    4962 scope.go:122] "RemoveContainer" containerID="818cb4a9e6555cfb76a34b6ac6fb6b2bd9e430b059d01e7e5a0659907bbdc053"
Oct 29 22:15:46.747476 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.747442    4962 scope.go:122] "RemoveContainer" containerID="818cb4a9e6555cfb76a34b6ac6fb6b2bd9e430b059d01e7e5a0659907bbdc053"
Oct 29 22:15:46.747926 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:15:46.747904    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"818cb4a9e6555cfb76a34b6ac6fb6b2bd9e430b059d01e7e5a0659907bbdc053\": container with ID starting with 818cb4a9e6555cfb76a34b6ac6fb6b2bd9e430b059d01e7e5a0659907bbdc053 not found: ID does not exist" containerID="818cb4a9e6555cfb76a34b6ac6fb6b2bd9e430b059d01e7e5a0659907bbdc053"
Oct 29 22:15:46.748022 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.747929    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"818cb4a9e6555cfb76a34b6ac6fb6b2bd9e430b059d01e7e5a0659907bbdc053"} err="failed to get container status \"818cb4a9e6555cfb76a34b6ac6fb6b2bd9e430b059d01e7e5a0659907bbdc053\": rpc error: code = NotFound desc = could not find container \"818cb4a9e6555cfb76a34b6ac6fb6b2bd9e430b059d01e7e5a0659907bbdc053\": container with ID starting with 818cb4a9e6555cfb76a34b6ac6fb6b2bd9e430b059d01e7e5a0659907bbdc053 not found: ID does not exist"
Oct 29 22:15:46.748022 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.747945    4962 scope.go:122] "RemoveContainer" containerID="5deab39450f38e650d846ff0ebd3c07ff1e0c246b2c66d6e46784f74bf0818d3"
Oct 29 22:15:46.756436 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.756417    4962 scope.go:122] "RemoveContainer" containerID="5deab39450f38e650d846ff0ebd3c07ff1e0c246b2c66d6e46784f74bf0818d3"
Oct 29 22:15:46.756670 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:15:46.756653    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5deab39450f38e650d846ff0ebd3c07ff1e0c246b2c66d6e46784f74bf0818d3\": container with ID starting with 5deab39450f38e650d846ff0ebd3c07ff1e0c246b2c66d6e46784f74bf0818d3 not found: ID does not exist" containerID="5deab39450f38e650d846ff0ebd3c07ff1e0c246b2c66d6e46784f74bf0818d3"
Oct 29 22:15:46.756713 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.756675    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5deab39450f38e650d846ff0ebd3c07ff1e0c246b2c66d6e46784f74bf0818d3"} err="failed to get container status \"5deab39450f38e650d846ff0ebd3c07ff1e0c246b2c66d6e46784f74bf0818d3\": rpc error: code = NotFound desc = could not find container \"5deab39450f38e650d846ff0ebd3c07ff1e0c246b2c66d6e46784f74bf0818d3\": container with ID starting with 5deab39450f38e650d846ff0ebd3c07ff1e0c246b2c66d6e46784f74bf0818d3 not found: ID does not exist"
Oct 29 22:15:46.785450 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.785423    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"]
Oct 29 22:15:46.796972 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.796951    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-2287s"]
Oct 29 22:15:46.835637 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.835613    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"]
Oct 29 22:15:46.839118 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.839092    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-crrsb"]
Oct 29 22:15:46.872025 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.872005    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"]
Oct 29 22:15:46.875516 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:46.875495    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f363a3-openshift-default-66498c7fb7-gg5nw"]
Oct 29 22:15:47.057756 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:47.057685    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9da507f9-8669-4b99-99de-8a7a83cd7850" path="/var/lib/kubelet/pods/9da507f9-8669-4b99-99de-8a7a83cd7850/volumes"
Oct 29 22:15:47.058121 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:47.058108    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4cda533-309f-46e1-8cad-7239630ac19d" path="/var/lib/kubelet/pods/b4cda533-309f-46e1-8cad-7239630ac19d/volumes"
Oct 29 22:15:47.058559 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:47.058545    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8c4fd64-8818-475c-b3d0-579054bcd76a" path="/var/lib/kubelet/pods/e8c4fd64-8818-475c-b3d0-579054bcd76a/volumes"
Oct 29 22:15:55.278571 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:55.278537    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg"]
Oct 29 22:15:55.279044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:55.278787    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg" podUID="dec9c57a-18ef-4c9c-8dd1-6876994b6578" containerName="maas-api" containerID="cri-o://7c44bb504d9f825be4e51db76182f550c3f9c33201d91d03bb5f28ea0a730fb8" gracePeriod=30
Oct 29 22:15:55.565180 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:55.565154    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg"
Oct 29 22:15:55.632885 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:55.632858    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dec9c57a-18ef-4c9c-8dd1-6876994b6578-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dec9c57a-18ef-4c9c-8dd1-6876994b6578-maas-api-tls\") pod \"dec9c57a-18ef-4c9c-8dd1-6876994b6578\" (UID: \"dec9c57a-18ef-4c9c-8dd1-6876994b6578\") "
Oct 29 22:15:55.633023 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:55.632943    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dec9c57a-18ef-4c9c-8dd1-6876994b6578-kube-api-access-ss44p\" (UniqueName: \"kubernetes.io/projected/dec9c57a-18ef-4c9c-8dd1-6876994b6578-kube-api-access-ss44p\") pod \"dec9c57a-18ef-4c9c-8dd1-6876994b6578\" (UID: \"dec9c57a-18ef-4c9c-8dd1-6876994b6578\") "
Oct 29 22:15:55.633023 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:55.633005    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dec9c57a-18ef-4c9c-8dd1-6876994b6578-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec9c57a-18ef-4c9c-8dd1-6876994b6578-metrics-certs\") pod \"dec9c57a-18ef-4c9c-8dd1-6876994b6578\" (UID: \"dec9c57a-18ef-4c9c-8dd1-6876994b6578\") "
Oct 29 22:15:55.633122 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:55.633044    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dec9c57a-18ef-4c9c-8dd1-6876994b6578-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dec9c57a-18ef-4c9c-8dd1-6876994b6578-openshift-service-ca\") pod \"dec9c57a-18ef-4c9c-8dd1-6876994b6578\" (UID: \"dec9c57a-18ef-4c9c-8dd1-6876994b6578\") "
Oct 29 22:15:55.633528 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:55.633495    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec9c57a-18ef-4c9c-8dd1-6876994b6578-openshift-service-ca" pod "dec9c57a-18ef-4c9c-8dd1-6876994b6578" (UID: "dec9c57a-18ef-4c9c-8dd1-6876994b6578"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:15:55.635328 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:55.635270    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec9c57a-18ef-4c9c-8dd1-6876994b6578-kube-api-access-ss44p" pod "dec9c57a-18ef-4c9c-8dd1-6876994b6578" (UID: "dec9c57a-18ef-4c9c-8dd1-6876994b6578"). InnerVolumeSpecName "kube-api-access-ss44p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:15:55.635690 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:55.635668    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dec9c57a-18ef-4c9c-8dd1-6876994b6578-metrics-certs" pod "dec9c57a-18ef-4c9c-8dd1-6876994b6578" (UID: "dec9c57a-18ef-4c9c-8dd1-6876994b6578"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:55.635770 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:55.635687    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dec9c57a-18ef-4c9c-8dd1-6876994b6578-maas-api-tls" pod "dec9c57a-18ef-4c9c-8dd1-6876994b6578" (UID: "dec9c57a-18ef-4c9c-8dd1-6876994b6578"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:15:55.737102 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:55.737065    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dec9c57a-18ef-4c9c-8dd1-6876994b6578-maas-api-tls\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:55.737406 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:55.737385    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss44p\" (UniqueName: \"kubernetes.io/projected/dec9c57a-18ef-4c9c-8dd1-6876994b6578-kube-api-access-ss44p\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:55.737527 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:55.737514    4962 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec9c57a-18ef-4c9c-8dd1-6876994b6578-metrics-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:55.737626 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:55.737614    4962 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dec9c57a-18ef-4c9c-8dd1-6876994b6578-openshift-service-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:15:55.817675 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:55.817566    4962 generic.go:356] "Generic (PLEG): container finished" podID="dec9c57a-18ef-4c9c-8dd1-6876994b6578" containerID="7c44bb504d9f825be4e51db76182f550c3f9c33201d91d03bb5f28ea0a730fb8" exitCode=0
Oct 29 22:15:55.817978 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:55.817702    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg"
Oct 29 22:15:55.818440 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:55.817722    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg" event={"ID":"dec9c57a-18ef-4c9c-8dd1-6876994b6578","Type":"ContainerDied","Data":"7c44bb504d9f825be4e51db76182f550c3f9c33201d91d03bb5f28ea0a730fb8"}
Oct 29 22:15:55.818676 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:55.818656    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg" event={"ID":"dec9c57a-18ef-4c9c-8dd1-6876994b6578","Type":"ContainerDied","Data":"b778034116c39b391c1a527b7571fca58d90f898db6514b5a0030be060839ac7"}
Oct 29 22:15:55.818808 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:55.818794    4962 scope.go:122] "RemoveContainer" containerID="7c44bb504d9f825be4e51db76182f550c3f9c33201d91d03bb5f28ea0a730fb8"
Oct 29 22:15:55.834228 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:55.834206    4962 scope.go:122] "RemoveContainer" containerID="7c44bb504d9f825be4e51db76182f550c3f9c33201d91d03bb5f28ea0a730fb8"
Oct 29 22:15:55.834519 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:15:55.834496    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c44bb504d9f825be4e51db76182f550c3f9c33201d91d03bb5f28ea0a730fb8\": container with ID starting with 7c44bb504d9f825be4e51db76182f550c3f9c33201d91d03bb5f28ea0a730fb8 not found: ID does not exist" containerID="7c44bb504d9f825be4e51db76182f550c3f9c33201d91d03bb5f28ea0a730fb8"
Oct 29 22:15:55.834619 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:55.834526    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c44bb504d9f825be4e51db76182f550c3f9c33201d91d03bb5f28ea0a730fb8"} err="failed to get container status \"7c44bb504d9f825be4e51db76182f550c3f9c33201d91d03bb5f28ea0a730fb8\": rpc error: code = NotFound desc = could not find container \"7c44bb504d9f825be4e51db76182f550c3f9c33201d91d03bb5f28ea0a730fb8\": container with ID starting with 7c44bb504d9f825be4e51db76182f550c3f9c33201d91d03bb5f28ea0a730fb8 not found: ID does not exist"
Oct 29 22:15:55.871858 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:55.871822    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg"]
Oct 29 22:15:55.882113 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:55.882086    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a9e67f-58597d488c-zxlzg"]
Oct 29 22:15:57.056144 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:15:57.056112    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dec9c57a-18ef-4c9c-8dd1-6876994b6578" path="/var/lib/kubelet/pods/dec9c57a-18ef-4c9c-8dd1-6876994b6578/volumes"
Oct 29 22:16:03.597210 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:03.597171    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"]
Oct 29 22:16:03.598171 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:03.598139    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn" podUID="bced75a3-37ba-4595-9dc0-b354574eba07" containerName="istio-proxy" containerID="cri-o://678eac811730ab90c261ea74bfe16e0c5ed7e0a918d54786252bd39a6f922e73" gracePeriod=30
Oct 29 22:16:03.631525 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:03.631483    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"]
Oct 29 22:16:03.631863 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:03.631817    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerName="istio-proxy" containerID="cri-o://81520d8cb39c5982bf8fb6f764297eabd7da42c56858f6fadfc97a5fc45cde0b" gracePeriod=30
Oct 29 22:16:03.641252 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:03.635918    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"]
Oct 29 22:16:03.641252 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:03.636132    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9" podUID="04500ffa-4da1-4d50-8fee-4e00ad03f4b0" containerName="istio-proxy" containerID="cri-o://8d4f497475dd9cbd8413f35b5f3f90cf13bdf6084173a78a31d0c07b31d7ff30" gracePeriod=30
Oct 29 22:16:04.316144 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316111    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4"]
Oct 29 22:16:04.316575 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316562    4962 cpu_manager.go:414] "containerMap: removing container" podUID="4a234504-3170-4ea9-aa0a-2d81ce2171a0" containerName="cleanup"
Oct 29 22:16:04.316634 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316576    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a234504-3170-4ea9-aa0a-2d81ce2171a0" containerName="cleanup"
Oct 29 22:16:04.316634 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316586    4962 cpu_manager.go:414] "containerMap: removing container" podUID="99bc070b-5d76-4cf5-b485-619e07383f53" containerName="cleanup"
Oct 29 22:16:04.316634 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316591    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99bc070b-5d76-4cf5-b485-619e07383f53" containerName="cleanup"
Oct 29 22:16:04.316634 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316598    4962 cpu_manager.go:414] "containerMap: removing container" podUID="99bc070b-5d76-4cf5-b485-619e07383f53" containerName="cleanup"
Oct 29 22:16:04.316634 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316604    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99bc070b-5d76-4cf5-b485-619e07383f53" containerName="cleanup"
Oct 29 22:16:04.316634 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316618    4962 cpu_manager.go:414] "containerMap: removing container" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main"
Oct 29 22:16:04.316634 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316623    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main"
Oct 29 22:16:04.316634 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316634    4962 cpu_manager.go:414] "containerMap: removing container" podUID="fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d" containerName="payload-processing"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316639    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d" containerName="payload-processing"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316647    4962 cpu_manager.go:414] "containerMap: removing container" podUID="ba71124e-36fa-41ea-bb8b-4f4375a1ac7f" containerName="payload-pre-processing"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316652    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba71124e-36fa-41ea-bb8b-4f4375a1ac7f" containerName="payload-pre-processing"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316663    4962 cpu_manager.go:414] "containerMap: removing container" podUID="4a234504-3170-4ea9-aa0a-2d81ce2171a0" containerName="cleanup"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316667    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a234504-3170-4ea9-aa0a-2d81ce2171a0" containerName="cleanup"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316673    4962 cpu_manager.go:414] "containerMap: removing container" podUID="f6631eff-5d71-483c-aac7-8d578274ce76" containerName="istio-proxy"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316677    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6631eff-5d71-483c-aac7-8d578274ce76" containerName="istio-proxy"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316685    4962 cpu_manager.go:414] "containerMap: removing container" podUID="dec9c57a-18ef-4c9c-8dd1-6876994b6578" containerName="maas-api"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316689    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dec9c57a-18ef-4c9c-8dd1-6876994b6578" containerName="maas-api"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316695    4962 cpu_manager.go:414] "containerMap: removing container" podUID="9da507f9-8669-4b99-99de-8a7a83cd7850" containerName="istio-proxy"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316700    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da507f9-8669-4b99-99de-8a7a83cd7850" containerName="istio-proxy"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316709    4962 cpu_manager.go:414] "containerMap: removing container" podUID="4a234504-3170-4ea9-aa0a-2d81ce2171a0" containerName="cleanup"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316713    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a234504-3170-4ea9-aa0a-2d81ce2171a0" containerName="cleanup"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316723    4962 cpu_manager.go:414] "containerMap: removing container" podUID="9762d8d0-a759-42ac-bc1b-c948adfa6f9d" containerName="istio-proxy"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316729    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9762d8d0-a759-42ac-bc1b-c948adfa6f9d" containerName="istio-proxy"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316734    4962 cpu_manager.go:414] "containerMap: removing container" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316739    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316747    4962 cpu_manager.go:414] "containerMap: removing container" podUID="b4cda533-309f-46e1-8cad-7239630ac19d" containerName="istio-proxy"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316751    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4cda533-309f-46e1-8cad-7239630ac19d" containerName="istio-proxy"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316764    4962 cpu_manager.go:414] "containerMap: removing container" podUID="4ef87fe2-429a-4393-8ddf-fa43e2dd9044" containerName="istio-proxy"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316768    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ef87fe2-429a-4393-8ddf-fa43e2dd9044" containerName="istio-proxy"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316775    4962 cpu_manager.go:414] "containerMap: removing container" podUID="e8c4fd64-8818-475c-b3d0-579054bcd76a" containerName="istio-proxy"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316780    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8c4fd64-8818-475c-b3d0-579054bcd76a" containerName="istio-proxy"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316844    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="9762d8d0-a759-42ac-bc1b-c948adfa6f9d" containerName="istio-proxy"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316861    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ac13470-5074-4ded-8f00-e98621700e54" containerName="main"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316868    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4cda533-309f-46e1-8cad-7239630ac19d" containerName="istio-proxy"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316877    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ef87fe2-429a-4393-8ddf-fa43e2dd9044" containerName="istio-proxy"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316883    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8c4fd64-8818-475c-b3d0-579054bcd76a" containerName="istio-proxy"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316890    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a234504-3170-4ea9-aa0a-2d81ce2171a0" containerName="cleanup"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316895    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="99bc070b-5d76-4cf5-b485-619e07383f53" containerName="cleanup"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316899    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="99bc070b-5d76-4cf5-b485-619e07383f53" containerName="cleanup"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316907    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e7bb25d-994d-4600-8ade-890853b6f0d3" containerName="main"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316914    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdc1ce1e-a65b-4e84-aa01-4dfc9780e14d" containerName="payload-processing"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316919    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba71124e-36fa-41ea-bb8b-4f4375a1ac7f" containerName="payload-pre-processing"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316925    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6631eff-5d71-483c-aac7-8d578274ce76" containerName="istio-proxy"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316931    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="dec9c57a-18ef-4c9c-8dd1-6876994b6578" containerName="maas-api"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316937    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da507f9-8669-4b99-99de-8a7a83cd7850" containerName="istio-proxy"
Oct 29 22:16:04.316964 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.316942    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a234504-3170-4ea9-aa0a-2d81ce2171a0" containerName="cleanup"
Oct 29 22:16:04.321490 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.321470    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4"
Oct 29 22:16:04.335771 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.335750    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:16:04.335883 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.335859    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:16:04.349470 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.349438    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4"]
Oct 29 22:16:04.370292 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.370265    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4067333-0059-4b1f-9f15-974985dba84a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4\" (UID: \"c4067333-0059-4b1f-9f15-974985dba84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4"
Oct 29 22:16:04.370442 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.370425    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bch52\" (UniqueName: \"kubernetes.io/projected/c4067333-0059-4b1f-9f15-974985dba84a-kube-api-access-bch52\") pod \"maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4\" (UID: \"c4067333-0059-4b1f-9f15-974985dba84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4"
Oct 29 22:16:04.370525 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.370459    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4067333-0059-4b1f-9f15-974985dba84a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4\" (UID: \"c4067333-0059-4b1f-9f15-974985dba84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4"
Oct 29 22:16:04.373002 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.372985    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:16:04.373103 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.373086    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:16:04.471524 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.471503    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4067333-0059-4b1f-9f15-974985dba84a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4\" (UID: \"c4067333-0059-4b1f-9f15-974985dba84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4"
Oct 29 22:16:04.471639 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.471554    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bch52\" (UniqueName: \"kubernetes.io/projected/c4067333-0059-4b1f-9f15-974985dba84a-kube-api-access-bch52\") pod \"maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4\" (UID: \"c4067333-0059-4b1f-9f15-974985dba84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4"
Oct 29 22:16:04.471639 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.471582    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4067333-0059-4b1f-9f15-974985dba84a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4\" (UID: \"c4067333-0059-4b1f-9f15-974985dba84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4"
Oct 29 22:16:04.472135 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.472116    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4067333-0059-4b1f-9f15-974985dba84a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4\" (UID: \"c4067333-0059-4b1f-9f15-974985dba84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4"
Oct 29 22:16:04.474016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.473997    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4067333-0059-4b1f-9f15-974985dba84a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4\" (UID: \"c4067333-0059-4b1f-9f15-974985dba84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4"
Oct 29 22:16:04.494150 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.494118    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bch52\" (UniqueName: \"kubernetes.io/projected/c4067333-0059-4b1f-9f15-974985dba84a-kube-api-access-bch52\") pod \"maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4\" (UID: \"c4067333-0059-4b1f-9f15-974985dba84a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4"
Oct 29 22:16:04.634455 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.634397    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4"
Oct 29 22:16:04.935612 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:04.935583    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4"]
Oct 29 22:16:04.937801 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:16:04.937776    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4067333_0059_4b1f_9f15_974985dba84a.slice/crio-21ce4feeb33bc823b3097574429ba2a9ffc0e189f27dcd393b9edf492702ab19 WatchSource:0}: Error finding container 21ce4feeb33bc823b3097574429ba2a9ffc0e189f27dcd393b9edf492702ab19: Status 404 returned error can't find the container with id 21ce4feeb33bc823b3097574429ba2a9ffc0e189f27dcd393b9edf492702ab19
Oct 29 22:16:05.918465 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:05.918432    4962 generic.go:356] "Generic (PLEG): container finished" podID="c4067333-0059-4b1f-9f15-974985dba84a" containerID="3c09e380ced313b63532b3e84feb23193d19c28957839e0933df3b8fd994748a" exitCode=0
Oct 29 22:16:05.918835 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:05.918488    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4" event={"ID":"c4067333-0059-4b1f-9f15-974985dba84a","Type":"ContainerDied","Data":"3c09e380ced313b63532b3e84feb23193d19c28957839e0933df3b8fd994748a"}
Oct 29 22:16:05.918835 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:05.918530    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4" event={"ID":"c4067333-0059-4b1f-9f15-974985dba84a","Type":"ContainerStarted","Data":"21ce4feeb33bc823b3097574429ba2a9ffc0e189f27dcd393b9edf492702ab19"}
Oct 29 22:16:06.624033 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:06.623996    4962 patch_prober.go:29] interesting pod/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:16:06.624202 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:06.624052    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn" podUID="bced75a3-37ba-4595-9dc0-b354574eba07" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:16:07.068926 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:07.068900    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4"
Oct 29 22:16:07.197345 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:07.197244    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4067333-0059-4b1f-9f15-974985dba84a-kube-api-access-bch52\" (UniqueName: \"kubernetes.io/projected/c4067333-0059-4b1f-9f15-974985dba84a-kube-api-access-bch52\") pod \"c4067333-0059-4b1f-9f15-974985dba84a\" (UID: \"c4067333-0059-4b1f-9f15-974985dba84a\") "
Oct 29 22:16:07.197345 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:07.197336    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4067333-0059-4b1f-9f15-974985dba84a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4067333-0059-4b1f-9f15-974985dba84a-maas-api-openshift-ca\") pod \"c4067333-0059-4b1f-9f15-974985dba84a\" (UID: \"c4067333-0059-4b1f-9f15-974985dba84a\") "
Oct 29 22:16:07.197583 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:07.197423    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4067333-0059-4b1f-9f15-974985dba84a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4067333-0059-4b1f-9f15-974985dba84a-maas-api-xks-ca\") pod \"c4067333-0059-4b1f-9f15-974985dba84a\" (UID: \"c4067333-0059-4b1f-9f15-974985dba84a\") "
Oct 29 22:16:07.197700 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:07.197677    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4067333-0059-4b1f-9f15-974985dba84a-maas-api-openshift-ca" pod "c4067333-0059-4b1f-9f15-974985dba84a" (UID: "c4067333-0059-4b1f-9f15-974985dba84a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:07.197958 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:07.197940    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4067333-0059-4b1f-9f15-974985dba84a-maas-api-openshift-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:07.199620 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:07.199596    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4067333-0059-4b1f-9f15-974985dba84a-kube-api-access-bch52" pod "c4067333-0059-4b1f-9f15-974985dba84a" (UID: "c4067333-0059-4b1f-9f15-974985dba84a"). InnerVolumeSpecName "kube-api-access-bch52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:07.199697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:07.199647    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4067333-0059-4b1f-9f15-974985dba84a-maas-api-xks-ca" pod "c4067333-0059-4b1f-9f15-974985dba84a" (UID: "c4067333-0059-4b1f-9f15-974985dba84a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:16:07.299429 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:07.299402    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bch52\" (UniqueName: \"kubernetes.io/projected/c4067333-0059-4b1f-9f15-974985dba84a-kube-api-access-bch52\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:07.299429 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:07.299428    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4067333-0059-4b1f-9f15-974985dba84a-maas-api-xks-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:07.930797 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:07.930765    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4"
Oct 29 22:16:07.930979 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:07.930794    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4" event={"ID":"c4067333-0059-4b1f-9f15-974985dba84a","Type":"ContainerDied","Data":"21ce4feeb33bc823b3097574429ba2a9ffc0e189f27dcd393b9edf492702ab19"}
Oct 29 22:16:07.930979 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:07.930844    4962 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21ce4feeb33bc823b3097574429ba2a9ffc0e189f27dcd393b9edf492702ab19"
Oct 29 22:16:08.943970 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:08.943714    4962 generic.go:356] "Generic (PLEG): container finished" podID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerID="81520d8cb39c5982bf8fb6f764297eabd7da42c56858f6fadfc97a5fc45cde0b" exitCode=0
Oct 29 22:16:08.943970 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:08.943844    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" event={"ID":"c584046d-f7a2-4cda-9965-e95d0dfe944d","Type":"ContainerDied","Data":"81520d8cb39c5982bf8fb6f764297eabd7da42c56858f6fadfc97a5fc45cde0b"}
Oct 29 22:16:08.956665 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:08.956345    4962 generic.go:356] "Generic (PLEG): container finished" podID="bced75a3-37ba-4595-9dc0-b354574eba07" containerID="678eac811730ab90c261ea74bfe16e0c5ed7e0a918d54786252bd39a6f922e73" exitCode=0
Oct 29 22:16:08.956665 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:08.956455    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn" event={"ID":"bced75a3-37ba-4595-9dc0-b354574eba07","Type":"ContainerDied","Data":"678eac811730ab90c261ea74bfe16e0c5ed7e0a918d54786252bd39a6f922e73"}
Oct 29 22:16:08.956665 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:08.956509    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn" event={"ID":"bced75a3-37ba-4595-9dc0-b354574eba07","Type":"ContainerDied","Data":"5b97bedba88180c27e5559e00f730a87a2958e1a744dbf9105a69b800bfb1798"}
Oct 29 22:16:08.956665 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:08.956536    4962 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b97bedba88180c27e5559e00f730a87a2958e1a744dbf9105a69b800bfb1798"
Oct 29 22:16:08.956665 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:08.956555    4962 scope.go:122] "RemoveContainer" containerID="5ab3ad1e4ae32a29b221951ad5682083847f64fb401f6a51f2a83afd4d835689"
Oct 29 22:16:08.961211 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:08.960980    4962 generic.go:356] "Generic (PLEG): container finished" podID="04500ffa-4da1-4d50-8fee-4e00ad03f4b0" containerID="8d4f497475dd9cbd8413f35b5f3f90cf13bdf6084173a78a31d0c07b31d7ff30" exitCode=0
Oct 29 22:16:08.961211 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:08.961076    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9" event={"ID":"04500ffa-4da1-4d50-8fee-4e00ad03f4b0","Type":"ContainerDied","Data":"8d4f497475dd9cbd8413f35b5f3f90cf13bdf6084173a78a31d0c07b31d7ff30"}
Oct 29 22:16:08.965755 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:08.965466    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:16:09.028892 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.027596    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-workload-socket\") pod \"bced75a3-37ba-4595-9dc0-b354574eba07\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") "
Oct 29 22:16:09.028892 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.027652    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bced75a3-37ba-4595-9dc0-b354574eba07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bced75a3-37ba-4595-9dc0-b354574eba07-istiod-ca-cert\") pod \"bced75a3-37ba-4595-9dc0-b354574eba07\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") "
Oct 29 22:16:09.028892 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.027683    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bced75a3-37ba-4595-9dc0-b354574eba07-istio-token\" (UniqueName: \"kubernetes.io/projected/bced75a3-37ba-4595-9dc0-b354574eba07-istio-token\") pod \"bced75a3-37ba-4595-9dc0-b354574eba07\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") "
Oct 29 22:16:09.028892 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.027739    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-istio-envoy\") pod \"bced75a3-37ba-4595-9dc0-b354574eba07\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") "
Oct 29 22:16:09.028892 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.027822    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-workload-certs\") pod \"bced75a3-37ba-4595-9dc0-b354574eba07\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") "
Oct 29 22:16:09.028892 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.027868    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bced75a3-37ba-4595-9dc0-b354574eba07-kube-api-access-jbj75\" (UniqueName: \"kubernetes.io/projected/bced75a3-37ba-4595-9dc0-b354574eba07-kube-api-access-jbj75\") pod \"bced75a3-37ba-4595-9dc0-b354574eba07\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") "
Oct 29 22:16:09.028892 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.027954    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-istio-data\") pod \"bced75a3-37ba-4595-9dc0-b354574eba07\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") "
Oct 29 22:16:09.028892 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.028068    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-credential-socket\") pod \"bced75a3-37ba-4595-9dc0-b354574eba07\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") "
Oct 29 22:16:09.028892 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.028106    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bced75a3-37ba-4595-9dc0-b354574eba07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bced75a3-37ba-4595-9dc0-b354574eba07-istio-podinfo\") pod \"bced75a3-37ba-4595-9dc0-b354574eba07\" (UID: \"bced75a3-37ba-4595-9dc0-b354574eba07\") "
Oct 29 22:16:09.032387 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.030764    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-workload-certs" pod "bced75a3-37ba-4595-9dc0-b354574eba07" (UID: "bced75a3-37ba-4595-9dc0-b354574eba07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.032387 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.031087    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-workload-socket" pod "bced75a3-37ba-4595-9dc0-b354574eba07" (UID: "bced75a3-37ba-4595-9dc0-b354574eba07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.032387 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.031441    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bced75a3-37ba-4595-9dc0-b354574eba07-istiod-ca-cert" pod "bced75a3-37ba-4595-9dc0-b354574eba07" (UID: "bced75a3-37ba-4595-9dc0-b354574eba07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:09.032387 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.031695    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-istio-data" pod "bced75a3-37ba-4595-9dc0-b354574eba07" (UID: "bced75a3-37ba-4595-9dc0-b354574eba07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.032387 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.032356    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-credential-socket" pod "bced75a3-37ba-4595-9dc0-b354574eba07" (UID: "bced75a3-37ba-4595-9dc0-b354574eba07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.131216 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.131113    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.131216 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.131147    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.131216 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.131162    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.131216 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.131177    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bced75a3-37ba-4595-9dc0-b354574eba07-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.131216 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.131191    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.203489 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.203455    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"]
Oct 29 22:16:09.203678 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.203511    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"]
Oct 29 22:16:09.291843 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.291580    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-istio-envoy" pod "bced75a3-37ba-4595-9dc0-b354574eba07" (UID: "bced75a3-37ba-4595-9dc0-b354574eba07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.300410 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.300363    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bced75a3-37ba-4595-9dc0-b354574eba07-istio-podinfo" pod "bced75a3-37ba-4595-9dc0-b354574eba07" (UID: "bced75a3-37ba-4595-9dc0-b354574eba07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:16:09.304624 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.304583    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bced75a3-37ba-4595-9dc0-b354574eba07-kube-api-access-jbj75" pod "bced75a3-37ba-4595-9dc0-b354574eba07" (UID: "bced75a3-37ba-4595-9dc0-b354574eba07"). InnerVolumeSpecName "kube-api-access-jbj75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:09.307016 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.306758    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:16:09.354097 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.354026    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbj75\" (UniqueName: \"kubernetes.io/projected/bced75a3-37ba-4595-9dc0-b354574eba07-kube-api-access-jbj75\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.354097 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.354057    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bced75a3-37ba-4595-9dc0-b354574eba07-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.354097 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.354073    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bced75a3-37ba-4595-9dc0-b354574eba07-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.376468 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.376418    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bced75a3-37ba-4595-9dc0-b354574eba07-istio-token" pod "bced75a3-37ba-4595-9dc0-b354574eba07" (UID: "bced75a3-37ba-4595-9dc0-b354574eba07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:09.476044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.455598    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-token\" (UniqueName: \"kubernetes.io/projected/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-token\") pod \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") "
Oct 29 22:16:09.476044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.455675    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-envoy\") pod \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") "
Oct 29 22:16:09.476044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.455735    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-data\") pod \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") "
Oct 29 22:16:09.476044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.455799    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-workload-socket\") pod \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") "
Oct 29 22:16:09.476044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.455841    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-workload-certs\") pod \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") "
Oct 29 22:16:09.476044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.455904    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-podinfo\") pod \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") "
Oct 29 22:16:09.476044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.455974    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istiod-ca-cert\") pod \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") "
Oct 29 22:16:09.476044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.456011    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-credential-socket\") pod \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") "
Oct 29 22:16:09.476044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.456053    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-kube-api-access-2bxt4\" (UniqueName: \"kubernetes.io/projected/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-kube-api-access-2bxt4\") pod \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\" (UID: \"04500ffa-4da1-4d50-8fee-4e00ad03f4b0\") "
Oct 29 22:16:09.476044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.456501    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bced75a3-37ba-4595-9dc0-b354574eba07-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.476044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.463618    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-workload-socket" pod "04500ffa-4da1-4d50-8fee-4e00ad03f4b0" (UID: "04500ffa-4da1-4d50-8fee-4e00ad03f4b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.476044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.470574    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-data" pod "04500ffa-4da1-4d50-8fee-4e00ad03f4b0" (UID: "04500ffa-4da1-4d50-8fee-4e00ad03f4b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.476044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.475182    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:16:09.476044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.475686    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istiod-ca-cert" pod "04500ffa-4da1-4d50-8fee-4e00ad03f4b0" (UID: "04500ffa-4da1-4d50-8fee-4e00ad03f4b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:09.476044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.475776    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-workload-certs" pod "04500ffa-4da1-4d50-8fee-4e00ad03f4b0" (UID: "04500ffa-4da1-4d50-8fee-4e00ad03f4b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.476044 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.475948    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-credential-socket" pod "04500ffa-4da1-4d50-8fee-4e00ad03f4b0" (UID: "04500ffa-4da1-4d50-8fee-4e00ad03f4b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.565343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.559755    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-workload-socket\") pod \"c584046d-f7a2-4cda-9965-e95d0dfe944d\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") "
Oct 29 22:16:09.565343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.559830    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-envoy\") pod \"c584046d-f7a2-4cda-9965-e95d0dfe944d\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") "
Oct 29 22:16:09.565343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.559872    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c584046d-f7a2-4cda-9965-e95d0dfe944d-kube-api-access-666vn\" (UniqueName: \"kubernetes.io/projected/c584046d-f7a2-4cda-9965-e95d0dfe944d-kube-api-access-666vn\") pod \"c584046d-f7a2-4cda-9965-e95d0dfe944d\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") "
Oct 29 22:16:09.565343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.559956    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-token\" (UniqueName: \"kubernetes.io/projected/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-token\") pod \"c584046d-f7a2-4cda-9965-e95d0dfe944d\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") "
Oct 29 22:16:09.565343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.559994    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-data\") pod \"c584046d-f7a2-4cda-9965-e95d0dfe944d\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") "
Oct 29 22:16:09.565343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.560064    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c584046d-f7a2-4cda-9965-e95d0dfe944d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c584046d-f7a2-4cda-9965-e95d0dfe944d-istiod-ca-cert\") pod \"c584046d-f7a2-4cda-9965-e95d0dfe944d\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") "
Oct 29 22:16:09.565343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.560132    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-workload-certs\") pod \"c584046d-f7a2-4cda-9965-e95d0dfe944d\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") "
Oct 29 22:16:09.565343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.560175    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-credential-socket\") pod \"c584046d-f7a2-4cda-9965-e95d0dfe944d\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") "
Oct 29 22:16:09.565343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.560227    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-podinfo\") pod \"c584046d-f7a2-4cda-9965-e95d0dfe944d\" (UID: \"c584046d-f7a2-4cda-9965-e95d0dfe944d\") "
Oct 29 22:16:09.565343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.560701    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.565343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.560724    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.565343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.560738    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.565343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.560753    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.565343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.560767    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.565343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.561558    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c584046d-f7a2-4cda-9965-e95d0dfe944d-istiod-ca-cert" pod "c584046d-f7a2-4cda-9965-e95d0dfe944d" (UID: "c584046d-f7a2-4cda-9965-e95d0dfe944d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:09.565343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.562014    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-data" pod "c584046d-f7a2-4cda-9965-e95d0dfe944d" (UID: "c584046d-f7a2-4cda-9965-e95d0dfe944d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.565343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.562673    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-workload-certs" pod "c584046d-f7a2-4cda-9965-e95d0dfe944d" (UID: "c584046d-f7a2-4cda-9965-e95d0dfe944d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.565343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.562876    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-credential-socket" pod "c584046d-f7a2-4cda-9965-e95d0dfe944d" (UID: "c584046d-f7a2-4cda-9965-e95d0dfe944d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.565343 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.563229    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-workload-socket" pod "c584046d-f7a2-4cda-9965-e95d0dfe944d" (UID: "c584046d-f7a2-4cda-9965-e95d0dfe944d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.587321 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.573402    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-token" pod "04500ffa-4da1-4d50-8fee-4e00ad03f4b0" (UID: "04500ffa-4da1-4d50-8fee-4e00ad03f4b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:09.587321 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.573808    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-envoy" pod "04500ffa-4da1-4d50-8fee-4e00ad03f4b0" (UID: "04500ffa-4da1-4d50-8fee-4e00ad03f4b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.587321 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.580525    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-kube-api-access-2bxt4" pod "04500ffa-4da1-4d50-8fee-4e00ad03f4b0" (UID: "04500ffa-4da1-4d50-8fee-4e00ad03f4b0"). InnerVolumeSpecName "kube-api-access-2bxt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:09.625346 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.608666    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-podinfo" pod "04500ffa-4da1-4d50-8fee-4e00ad03f4b0" (UID: "04500ffa-4da1-4d50-8fee-4e00ad03f4b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:16:09.713643 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.663234    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.713643 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.663262    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.713643 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.663280    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.713643 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.663312    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.713643 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.663327    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bxt4\" (UniqueName: \"kubernetes.io/projected/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-kube-api-access-2bxt4\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.713643 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.663341    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.713643 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.663357    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.713643 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.663370    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04500ffa-4da1-4d50-8fee-4e00ad03f4b0-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.713643 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.663382    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c584046d-f7a2-4cda-9965-e95d0dfe944d-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.747045 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.746964    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-envoy" pod "c584046d-f7a2-4cda-9965-e95d0dfe944d" (UID: "c584046d-f7a2-4cda-9965-e95d0dfe944d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:09.747366 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.747329    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-token" pod "c584046d-f7a2-4cda-9965-e95d0dfe944d" (UID: "c584046d-f7a2-4cda-9965-e95d0dfe944d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:09.756313 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.749375    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-podinfo" pod "c584046d-f7a2-4cda-9965-e95d0dfe944d" (UID: "c584046d-f7a2-4cda-9965-e95d0dfe944d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:16:09.788448 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.772967    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.788448 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.772993    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.788448 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.773007    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c584046d-f7a2-4cda-9965-e95d0dfe944d-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.795586 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.795547    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c584046d-f7a2-4cda-9965-e95d0dfe944d-kube-api-access-666vn" pod "c584046d-f7a2-4cda-9965-e95d0dfe944d" (UID: "c584046d-f7a2-4cda-9965-e95d0dfe944d"). InnerVolumeSpecName "kube-api-access-666vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:09.876031 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:09.875984    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-666vn\" (UniqueName: \"kubernetes.io/projected/c584046d-f7a2-4cda-9965-e95d0dfe944d-kube-api-access-666vn\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:09.954337 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:16:09.948959    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:16:10.076827 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:10.076731    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p" event={"ID":"c584046d-f7a2-4cda-9965-e95d0dfe944d","Type":"ContainerDied","Data":"07d8326d905e4fa4cc4ff0a6945d9f78bec2e0a269fdc6018ca995af53979bf1"}
Oct 29 22:16:10.081900 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:10.081873    4962 scope.go:122] "RemoveContainer" containerID="81520d8cb39c5982bf8fb6f764297eabd7da42c56858f6fadfc97a5fc45cde0b"
Oct 29 22:16:10.083023 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:10.082132    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"
Oct 29 22:16:10.083204 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:10.077407    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"
Oct 29 22:16:10.089059 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:10.089018    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9" event={"ID":"04500ffa-4da1-4d50-8fee-4e00ad03f4b0","Type":"ContainerDied","Data":"80e1b63bd97e70f8b321f1a094323d5024068f8094ad6851a9b03b66b5fdb2a8"}
Oct 29 22:16:10.090452 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:10.090433    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"
Oct 29 22:16:10.140207 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:10.140185    4962 scope.go:122] "RemoveContainer" containerID="8d4f497475dd9cbd8413f35b5f3f90cf13bdf6084173a78a31d0c07b31d7ff30"
Oct 29 22:16:10.216307 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:10.216253    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"]
Oct 29 22:16:10.221130 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:10.221096    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-j2hfn"]
Oct 29 22:16:10.252832 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:10.252800    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"]
Oct 29 22:16:10.257045 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:10.257016    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-mfzs9"]
Oct 29 22:16:10.288910 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:10.288881    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"]
Oct 29 22:16:10.294386 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:10.294361    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a9e67f-openshift-default-f9595bbb9-6xx7p"]
Oct 29 22:16:11.078970 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:11.078931    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04500ffa-4da1-4d50-8fee-4e00ad03f4b0" path="/var/lib/kubelet/pods/04500ffa-4da1-4d50-8fee-4e00ad03f4b0/volumes"
Oct 29 22:16:11.079634 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:11.079614    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bced75a3-37ba-4595-9dc0-b354574eba07" path="/var/lib/kubelet/pods/bced75a3-37ba-4595-9dc0-b354574eba07/volumes"
Oct 29 22:16:11.080224 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:11.080203    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" path="/var/lib/kubelet/pods/c584046d-f7a2-4cda-9965-e95d0dfe944d/volumes"
Oct 29 22:16:14.750802 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:14.750694    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr"]
Oct 29 22:16:14.782060 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:14.782016    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd"]
Oct 29 22:16:17.208778 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.208736    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fe3d91f-2z5j9"]
Oct 29 22:16:17.209181 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.209151    4962 cpu_manager.go:414] "containerMap: removing container" podUID="99bc070b-5d76-4cf5-b485-619e07383f53" containerName="cleanup"
Oct 29 22:16:17.209181 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.209163    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99bc070b-5d76-4cf5-b485-619e07383f53" containerName="cleanup"
Oct 29 22:16:17.209181 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.209173    4962 cpu_manager.go:414] "containerMap: removing container" podUID="04500ffa-4da1-4d50-8fee-4e00ad03f4b0" containerName="istio-proxy"
Oct 29 22:16:17.209181 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.209178    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04500ffa-4da1-4d50-8fee-4e00ad03f4b0" containerName="istio-proxy"
Oct 29 22:16:17.209356 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.209196    4962 cpu_manager.go:414] "containerMap: removing container" podUID="bced75a3-37ba-4595-9dc0-b354574eba07" containerName="istio-proxy"
Oct 29 22:16:17.209356 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.209202    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bced75a3-37ba-4595-9dc0-b354574eba07" containerName="istio-proxy"
Oct 29 22:16:17.209356 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.209212    4962 cpu_manager.go:414] "containerMap: removing container" podUID="bced75a3-37ba-4595-9dc0-b354574eba07" containerName="istio-proxy"
Oct 29 22:16:17.209356 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.209216    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bced75a3-37ba-4595-9dc0-b354574eba07" containerName="istio-proxy"
Oct 29 22:16:17.209356 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.209224    4962 cpu_manager.go:414] "containerMap: removing container" podUID="c4067333-0059-4b1f-9f15-974985dba84a" containerName="revoke-keys"
Oct 29 22:16:17.209356 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.209228    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4067333-0059-4b1f-9f15-974985dba84a" containerName="revoke-keys"
Oct 29 22:16:17.209356 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.209245    4962 cpu_manager.go:414] "containerMap: removing container" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerName="istio-proxy"
Oct 29 22:16:17.209356 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.209250    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerName="istio-proxy"
Oct 29 22:16:17.209356 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.209323    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="bced75a3-37ba-4595-9dc0-b354574eba07" containerName="istio-proxy"
Oct 29 22:16:17.209356 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.209333    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="bced75a3-37ba-4595-9dc0-b354574eba07" containerName="istio-proxy"
Oct 29 22:16:17.209356 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.209340    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4067333-0059-4b1f-9f15-974985dba84a" containerName="revoke-keys"
Oct 29 22:16:17.209356 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.209345    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a234504-3170-4ea9-aa0a-2d81ce2171a0" containerName="cleanup"
Oct 29 22:16:17.209356 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.209353    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="c584046d-f7a2-4cda-9965-e95d0dfe944d" containerName="istio-proxy"
Oct 29 22:16:17.209356 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.209360    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="99bc070b-5d76-4cf5-b485-619e07383f53" containerName="cleanup"
Oct 29 22:16:17.209775 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.209366    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="04500ffa-4da1-4d50-8fee-4e00ad03f4b0" containerName="istio-proxy"
Oct 29 22:16:17.212438 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.212415    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe3d91f-2z5j9"
Oct 29 22:16:17.229718 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.229698    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fwpnh\""
Oct 29 22:16:17.241618 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.241597    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:16:17.241734 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.241644    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:16:17.244248 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.244221    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fe3d91f-2z5j9"]
Oct 29 22:16:17.286906 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.286883    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adbf3c9f-a2e2-49b9-bef2-a9a0847b0374-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fe3d91f-2z5j9\" (UID: \"adbf3c9f-a2e2-49b9-bef2-a9a0847b0374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe3d91f-2z5j9"
Oct 29 22:16:17.287030 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.286916    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtb4s\" (UniqueName: \"kubernetes.io/projected/adbf3c9f-a2e2-49b9-bef2-a9a0847b0374-kube-api-access-qtb4s\") pod \"maas-api-revoke-sub-9fe3d91f-2z5j9\" (UID: \"adbf3c9f-a2e2-49b9-bef2-a9a0847b0374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe3d91f-2z5j9"
Oct 29 22:16:17.287030 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.286942    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adbf3c9f-a2e2-49b9-bef2-a9a0847b0374-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fe3d91f-2z5j9\" (UID: \"adbf3c9f-a2e2-49b9-bef2-a9a0847b0374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe3d91f-2z5j9"
Oct 29 22:16:17.330007 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.329987    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:16:17.387624 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.387600    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adbf3c9f-a2e2-49b9-bef2-a9a0847b0374-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fe3d91f-2z5j9\" (UID: \"adbf3c9f-a2e2-49b9-bef2-a9a0847b0374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe3d91f-2z5j9"
Oct 29 22:16:17.387733 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.387633    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtb4s\" (UniqueName: \"kubernetes.io/projected/adbf3c9f-a2e2-49b9-bef2-a9a0847b0374-kube-api-access-qtb4s\") pod \"maas-api-revoke-sub-9fe3d91f-2z5j9\" (UID: \"adbf3c9f-a2e2-49b9-bef2-a9a0847b0374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe3d91f-2z5j9"
Oct 29 22:16:17.387733 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.387657    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adbf3c9f-a2e2-49b9-bef2-a9a0847b0374-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fe3d91f-2z5j9\" (UID: \"adbf3c9f-a2e2-49b9-bef2-a9a0847b0374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe3d91f-2z5j9"
Oct 29 22:16:17.388233 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.388211    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adbf3c9f-a2e2-49b9-bef2-a9a0847b0374-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fe3d91f-2z5j9\" (UID: \"adbf3c9f-a2e2-49b9-bef2-a9a0847b0374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe3d91f-2z5j9"
Oct 29 22:16:17.389929 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.389905    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adbf3c9f-a2e2-49b9-bef2-a9a0847b0374-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fe3d91f-2z5j9\" (UID: \"adbf3c9f-a2e2-49b9-bef2-a9a0847b0374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe3d91f-2z5j9"
Oct 29 22:16:17.409877 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.409849    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtb4s\" (UniqueName: \"kubernetes.io/projected/adbf3c9f-a2e2-49b9-bef2-a9a0847b0374-kube-api-access-qtb4s\") pod \"maas-api-revoke-sub-9fe3d91f-2z5j9\" (UID: \"adbf3c9f-a2e2-49b9-bef2-a9a0847b0374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe3d91f-2z5j9"
Oct 29 22:16:17.525559 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.525479    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe3d91f-2z5j9"
Oct 29 22:16:17.828751 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:17.828725    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fe3d91f-2z5j9"]
Oct 29 22:16:17.830914 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:16:17.830883    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadbf3c9f_a2e2_49b9_bef2_a9a0847b0374.slice/crio-a22082f4ab026d931a5db8375fe9db687f42d367342fbbe7c268375e8003106e WatchSource:0}: Error finding container a22082f4ab026d931a5db8375fe9db687f42d367342fbbe7c268375e8003106e: Status 404 returned error can't find the container with id a22082f4ab026d931a5db8375fe9db687f42d367342fbbe7c268375e8003106e
Oct 29 22:16:18.167277 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:18.167244    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe3d91f-2z5j9" event={"ID":"adbf3c9f-a2e2-49b9-bef2-a9a0847b0374","Type":"ContainerStarted","Data":"55ecac240a296223e7b111741cc28a24f65f3fb55c6322e6f8aab0ad7b1c899e"}
Oct 29 22:16:18.167409 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:18.167313    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe3d91f-2z5j9" event={"ID":"adbf3c9f-a2e2-49b9-bef2-a9a0847b0374","Type":"ContainerStarted","Data":"a22082f4ab026d931a5db8375fe9db687f42d367342fbbe7c268375e8003106e"}
Oct 29 22:16:18.209840 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:18.209796    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe3d91f-2z5j9" podStartSLOduration=1.2097811219999999 podStartE2EDuration="1.209781122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:16:18.208141619 +0000 UTC m=+1799.281098582" watchObservedRunningTime="2026-10-29 22:16:18.209781122 +0000 UTC m=+1799.282738086"
Oct 29 22:16:19.173848 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:19.173814    4962 generic.go:356] "Generic (PLEG): container finished" podID="adbf3c9f-a2e2-49b9-bef2-a9a0847b0374" containerID="55ecac240a296223e7b111741cc28a24f65f3fb55c6322e6f8aab0ad7b1c899e" exitCode=0
Oct 29 22:16:19.174042 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:19.173857    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe3d91f-2z5j9" event={"ID":"adbf3c9f-a2e2-49b9-bef2-a9a0847b0374","Type":"ContainerDied","Data":"55ecac240a296223e7b111741cc28a24f65f3fb55c6322e6f8aab0ad7b1c899e"}
Oct 29 22:16:19.711369 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:19.711336    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/2.log"
Oct 29 22:16:19.712619 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:19.712596    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/2.log"
Oct 29 22:16:19.718894 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:19.718877    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74bdq_259e6b95-d64c-4101-871f-4a933d4ac38f/ovn-acl-logging/0.log"
Oct 29 22:16:19.719907 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:19.719881    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-74bdq_259e6b95-d64c-4101-871f-4a933d4ac38f/ovn-acl-logging/0.log"
Oct 29 22:16:19.751758 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:19.751715    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr" podUID="d37a81f7-dff2-429d-8dd1-c5d626299df8" containerName="payload-processing" containerID="cri-o://4092a0b64316186a01c4436afa0756c378079bc99d5911cd76886087054c1c2d" gracePeriod=25
Oct 29 22:16:19.782750 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:19.782713    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd" podUID="cee4891f-98dd-4ea8-a470-0f85ea62b849" containerName="payload-pre-processing" containerID="cri-o://d541e0eb15d76e65530d3b4835498b30cd423ad44a84e9c1cfd1d1f2e017be85" gracePeriod=25
Oct 29 22:16:20.313539 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:20.313516    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe3d91f-2z5j9"
Oct 29 22:16:20.414535 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:20.414510    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adbf3c9f-a2e2-49b9-bef2-a9a0847b0374-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adbf3c9f-a2e2-49b9-bef2-a9a0847b0374-maas-api-openshift-ca\") pod \"adbf3c9f-a2e2-49b9-bef2-a9a0847b0374\" (UID: \"adbf3c9f-a2e2-49b9-bef2-a9a0847b0374\") "
Oct 29 22:16:20.414666 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:20.414564    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adbf3c9f-a2e2-49b9-bef2-a9a0847b0374-kube-api-access-qtb4s\" (UniqueName: \"kubernetes.io/projected/adbf3c9f-a2e2-49b9-bef2-a9a0847b0374-kube-api-access-qtb4s\") pod \"adbf3c9f-a2e2-49b9-bef2-a9a0847b0374\" (UID: \"adbf3c9f-a2e2-49b9-bef2-a9a0847b0374\") "
Oct 29 22:16:20.414711 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:20.414681    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adbf3c9f-a2e2-49b9-bef2-a9a0847b0374-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adbf3c9f-a2e2-49b9-bef2-a9a0847b0374-maas-api-xks-ca\") pod \"adbf3c9f-a2e2-49b9-bef2-a9a0847b0374\" (UID: \"adbf3c9f-a2e2-49b9-bef2-a9a0847b0374\") "
Oct 29 22:16:20.414842 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:20.414821    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adbf3c9f-a2e2-49b9-bef2-a9a0847b0374-maas-api-openshift-ca" pod "adbf3c9f-a2e2-49b9-bef2-a9a0847b0374" (UID: "adbf3c9f-a2e2-49b9-bef2-a9a0847b0374"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:20.414972 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:20.414954    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adbf3c9f-a2e2-49b9-bef2-a9a0847b0374-maas-api-openshift-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:20.416615 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:20.416590    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbf3c9f-a2e2-49b9-bef2-a9a0847b0374-kube-api-access-qtb4s" pod "adbf3c9f-a2e2-49b9-bef2-a9a0847b0374" (UID: "adbf3c9f-a2e2-49b9-bef2-a9a0847b0374"). InnerVolumeSpecName "kube-api-access-qtb4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:20.416724 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:20.416710    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adbf3c9f-a2e2-49b9-bef2-a9a0847b0374-maas-api-xks-ca" pod "adbf3c9f-a2e2-49b9-bef2-a9a0847b0374" (UID: "adbf3c9f-a2e2-49b9-bef2-a9a0847b0374"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:16:20.516339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:20.516254    4962 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adbf3c9f-a2e2-49b9-bef2-a9a0847b0374-maas-api-xks-ca\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:20.516339 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:20.516278    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtb4s\" (UniqueName: \"kubernetes.io/projected/adbf3c9f-a2e2-49b9-bef2-a9a0847b0374-kube-api-access-qtb4s\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:21.186689 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:21.186665    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe3d91f-2z5j9"
Oct 29 22:16:21.187077 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:21.186680    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fe3d91f-2z5j9" event={"ID":"adbf3c9f-a2e2-49b9-bef2-a9a0847b0374","Type":"ContainerDied","Data":"a22082f4ab026d931a5db8375fe9db687f42d367342fbbe7c268375e8003106e"}
Oct 29 22:16:21.187077 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:21.186727    4962 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a22082f4ab026d931a5db8375fe9db687f42d367342fbbe7c268375e8003106e"
Oct 29 22:16:23.210315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:23.210263    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr" podUID="d37a81f7-dff2-429d-8dd1-c5d626299df8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:16:23.210315 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:23.210278    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd" podUID="cee4891f-98dd-4ea8-a470-0f85ea62b849" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:16:24.270311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:24.266496    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" containerID="cri-o://cd94d2fe13aa040aae836ba36a81875aaaef0ff3c3c4c8dcf88a7bd7e8fe5621" gracePeriod=45
Oct 29 22:16:24.316311 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:24.315458    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" containerID="cri-o://780fb9a873de22c245f9980b855ad6f272ca0f63bc80a67e728adfdecdb420e3" gracePeriod=45
Oct 29 22:16:24.422562 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:24.422509    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": EOF"
Oct 29 22:16:24.712826 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:24.712696    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": dial tcp 10.132.0.73:8000: connect: connection refused"
Oct 29 22:16:25.227780 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.227721    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:16:25.229109 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.229082    4962 generic.go:356] "Generic (PLEG): container finished" podID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerID="780fb9a873de22c245f9980b855ad6f272ca0f63bc80a67e728adfdecdb420e3" exitCode=0
Oct 29 22:16:25.229227 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.229192    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" event={"ID":"b9303c60-e8b1-4f7c-841f-811099ba8660","Type":"ContainerDied","Data":"780fb9a873de22c245f9980b855ad6f272ca0f63bc80a67e728adfdecdb420e3"}
Oct 29 22:16:25.230962 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.230937    4962 generic.go:356] "Generic (PLEG): container finished" podID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerID="cd94d2fe13aa040aae836ba36a81875aaaef0ff3c3c4c8dcf88a7bd7e8fe5621" exitCode=0
Oct 29 22:16:25.231068 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.231038    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" event={"ID":"e609098d-a77f-45bd-b87b-7e579dd0ed73","Type":"ContainerDied","Data":"cd94d2fe13aa040aae836ba36a81875aaaef0ff3c3c4c8dcf88a7bd7e8fe5621"}
Oct 29 22:16:25.231068 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.231064    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v" event={"ID":"e609098d-a77f-45bd-b87b-7e579dd0ed73","Type":"ContainerDied","Data":"ceb1b08e912b699c61f247d3d5991ab00853317962572048a733c9717908bdcb"}
Oct 29 22:16:25.231193 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.231086    4962 scope.go:122] "RemoveContainer" containerID="cd94d2fe13aa040aae836ba36a81875aaaef0ff3c3c4c8dcf88a7bd7e8fe5621"
Oct 29 22:16:25.231256 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.231231    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"
Oct 29 22:16:25.244536 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.244420    4962 scope.go:122] "RemoveContainer" containerID="cd94d2fe13aa040aae836ba36a81875aaaef0ff3c3c4c8dcf88a7bd7e8fe5621"
Oct 29 22:16:25.244703 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:16:25.244679    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd94d2fe13aa040aae836ba36a81875aaaef0ff3c3c4c8dcf88a7bd7e8fe5621\": container with ID starting with cd94d2fe13aa040aae836ba36a81875aaaef0ff3c3c4c8dcf88a7bd7e8fe5621 not found: ID does not exist" containerID="cd94d2fe13aa040aae836ba36a81875aaaef0ff3c3c4c8dcf88a7bd7e8fe5621"
Oct 29 22:16:25.244777 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.244710    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd94d2fe13aa040aae836ba36a81875aaaef0ff3c3c4c8dcf88a7bd7e8fe5621"} err="failed to get container status \"cd94d2fe13aa040aae836ba36a81875aaaef0ff3c3c4c8dcf88a7bd7e8fe5621\": rpc error: code = NotFound desc = could not find container \"cd94d2fe13aa040aae836ba36a81875aaaef0ff3c3c4c8dcf88a7bd7e8fe5621\": container with ID starting with cd94d2fe13aa040aae836ba36a81875aaaef0ff3c3c4c8dcf88a7bd7e8fe5621 not found: ID does not exist"
Oct 29 22:16:25.247272 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.247252    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:16:25.304398 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.304367    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e609098d-a77f-45bd-b87b-7e579dd0ed73-tls-certs\" (UniqueName: \"kubernetes.io/secret/e609098d-a77f-45bd-b87b-7e579dd0ed73-tls-certs\") pod \"e609098d-a77f-45bd-b87b-7e579dd0ed73\" (UID: \"e609098d-a77f-45bd-b87b-7e579dd0ed73\") "
Oct 29 22:16:25.304841 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.304416    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-model-cache\") pod \"e609098d-a77f-45bd-b87b-7e579dd0ed73\" (UID: \"e609098d-a77f-45bd-b87b-7e579dd0ed73\") "
Oct 29 22:16:25.304841 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.304459    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-home\" (UniqueName: \"kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-home\") pod \"e609098d-a77f-45bd-b87b-7e579dd0ed73\" (UID: \"e609098d-a77f-45bd-b87b-7e579dd0ed73\") "
Oct 29 22:16:25.304841 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.304499    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-home\" (UniqueName: \"kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-home\") pod \"b9303c60-e8b1-4f7c-841f-811099ba8660\" (UID: \"b9303c60-e8b1-4f7c-841f-811099ba8660\") "
Oct 29 22:16:25.304841 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.304530    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-tmp-dir\") pod \"e609098d-a77f-45bd-b87b-7e579dd0ed73\" (UID: \"e609098d-a77f-45bd-b87b-7e579dd0ed73\") "
Oct 29 22:16:25.304841 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.304592    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-dshm\") pod \"e609098d-a77f-45bd-b87b-7e579dd0ed73\" (UID: \"e609098d-a77f-45bd-b87b-7e579dd0ed73\") "
Oct 29 22:16:25.304841 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.304634    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-tmp-dir\") pod \"b9303c60-e8b1-4f7c-841f-811099ba8660\" (UID: \"b9303c60-e8b1-4f7c-841f-811099ba8660\") "
Oct 29 22:16:25.304841 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.304674    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-dshm\") pod \"b9303c60-e8b1-4f7c-841f-811099ba8660\" (UID: \"b9303c60-e8b1-4f7c-841f-811099ba8660\") "
Oct 29 22:16:25.304841 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.304705    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9303c60-e8b1-4f7c-841f-811099ba8660-kube-api-access-9c4ck\" (UniqueName: \"kubernetes.io/projected/b9303c60-e8b1-4f7c-841f-811099ba8660-kube-api-access-9c4ck\") pod \"b9303c60-e8b1-4f7c-841f-811099ba8660\" (UID: \"b9303c60-e8b1-4f7c-841f-811099ba8660\") "
Oct 29 22:16:25.304841 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.304741    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e609098d-a77f-45bd-b87b-7e579dd0ed73-kube-api-access-nfvjq\" (UniqueName: \"kubernetes.io/projected/e609098d-a77f-45bd-b87b-7e579dd0ed73-kube-api-access-nfvjq\") pod \"e609098d-a77f-45bd-b87b-7e579dd0ed73\" (UID: \"e609098d-a77f-45bd-b87b-7e579dd0ed73\") "
Oct 29 22:16:25.304841 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.304743    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-home" pod "e609098d-a77f-45bd-b87b-7e579dd0ed73" (UID: "e609098d-a77f-45bd-b87b-7e579dd0ed73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:25.304841 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.304753    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-model-cache" pod "e609098d-a77f-45bd-b87b-7e579dd0ed73" (UID: "e609098d-a77f-45bd-b87b-7e579dd0ed73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:25.304841 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.304774    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-model-cache\") pod \"b9303c60-e8b1-4f7c-841f-811099ba8660\" (UID: \"b9303c60-e8b1-4f7c-841f-811099ba8660\") "
Oct 29 22:16:25.304841 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.304823    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9303c60-e8b1-4f7c-841f-811099ba8660-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9303c60-e8b1-4f7c-841f-811099ba8660-tls-certs\") pod \"b9303c60-e8b1-4f7c-841f-811099ba8660\" (UID: \"b9303c60-e8b1-4f7c-841f-811099ba8660\") "
Oct 29 22:16:25.305716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.304965    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-tmp-dir" pod "e609098d-a77f-45bd-b87b-7e579dd0ed73" (UID: "e609098d-a77f-45bd-b87b-7e579dd0ed73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:25.305716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.304984    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-home" pod "b9303c60-e8b1-4f7c-841f-811099ba8660" (UID: "b9303c60-e8b1-4f7c-841f-811099ba8660"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:25.305716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.305200    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-tmp-dir" pod "b9303c60-e8b1-4f7c-841f-811099ba8660" (UID: "b9303c60-e8b1-4f7c-841f-811099ba8660"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:25.305716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.305326    4962 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-model-cache\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:25.305716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.305346    4962 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-home\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:25.305716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.305359    4962 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-home\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:25.305716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.305401    4962 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-tmp-dir\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:25.305716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.305416    4962 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-tmp-dir\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:25.305716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.305439    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-model-cache" pod "b9303c60-e8b1-4f7c-841f-811099ba8660" (UID: "b9303c60-e8b1-4f7c-841f-811099ba8660"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:25.306989 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.306965    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e609098d-a77f-45bd-b87b-7e579dd0ed73-tls-certs" pod "e609098d-a77f-45bd-b87b-7e579dd0ed73" (UID: "e609098d-a77f-45bd-b87b-7e579dd0ed73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:16:25.307260 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.307232    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-dshm" pod "b9303c60-e8b1-4f7c-841f-811099ba8660" (UID: "b9303c60-e8b1-4f7c-841f-811099ba8660"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:25.307383 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.307267    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9303c60-e8b1-4f7c-841f-811099ba8660-tls-certs" pod "b9303c60-e8b1-4f7c-841f-811099ba8660" (UID: "b9303c60-e8b1-4f7c-841f-811099ba8660"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:16:25.307587 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.307570    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9303c60-e8b1-4f7c-841f-811099ba8660-kube-api-access-9c4ck" pod "b9303c60-e8b1-4f7c-841f-811099ba8660" (UID: "b9303c60-e8b1-4f7c-841f-811099ba8660"). InnerVolumeSpecName "kube-api-access-9c4ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:25.307640 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.307595    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e609098d-a77f-45bd-b87b-7e579dd0ed73-kube-api-access-nfvjq" pod "e609098d-a77f-45bd-b87b-7e579dd0ed73" (UID: "e609098d-a77f-45bd-b87b-7e579dd0ed73"). InnerVolumeSpecName "kube-api-access-nfvjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:25.308427 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.308407    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-dshm" pod "e609098d-a77f-45bd-b87b-7e579dd0ed73" (UID: "e609098d-a77f-45bd-b87b-7e579dd0ed73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:25.405886 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.405854    4962 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e609098d-a77f-45bd-b87b-7e579dd0ed73-dshm\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:25.405886 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.405884    4962 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-dshm\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:25.406045 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.405893    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c4ck\" (UniqueName: \"kubernetes.io/projected/b9303c60-e8b1-4f7c-841f-811099ba8660-kube-api-access-9c4ck\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:25.406045 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.405903    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfvjq\" (UniqueName: \"kubernetes.io/projected/e609098d-a77f-45bd-b87b-7e579dd0ed73-kube-api-access-nfvjq\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:25.406045 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.405913    4962 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9303c60-e8b1-4f7c-841f-811099ba8660-model-cache\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:25.406045 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.405921    4962 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9303c60-e8b1-4f7c-841f-811099ba8660-tls-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:25.406045 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.405929    4962 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e609098d-a77f-45bd-b87b-7e579dd0ed73-tls-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:25.574561 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.574532    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"]
Oct 29 22:16:25.578501 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:25.578476    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8a9e5f/simulator-w3-8a9e5f-kserve-7944546ccb-h7t9v"]
Oct 29 22:16:26.200563 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:26.200526    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:16:26.200720 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:26.200613    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:16:26.238741 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:26.238716    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t" event={"ID":"b9303c60-e8b1-4f7c-841f-811099ba8660","Type":"ContainerDied","Data":"7a2f4f22fd9641caa081aec3a6d4f798d28e7411f1fc5dd55a07232a05e8ea37"}
Oct 29 22:16:26.238741 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:26.238737    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"
Oct 29 22:16:26.238741 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:26.238752    4962 scope.go:122] "RemoveContainer" containerID="780fb9a873de22c245f9980b855ad6f272ca0f63bc80a67e728adfdecdb420e3"
Oct 29 22:16:26.290060 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:26.290028    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"]
Oct 29 22:16:26.301659 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:26.301636    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8a9e5f/distinct-w3-8a9e5f-kserve-6bd76b5984-bzm7t"]
Oct 29 22:16:27.056350 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:27.056319    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" path="/var/lib/kubelet/pods/b9303c60-e8b1-4f7c-841f-811099ba8660/volumes"
Oct 29 22:16:27.056732 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:27.056710    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" path="/var/lib/kubelet/pods/e609098d-a77f-45bd-b87b-7e579dd0ed73/volumes"
Oct 29 22:16:27.209494 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:27.209467    4962 scope.go:122] "RemoveContainer" containerID="e817669945643f3f5e7efa4e1c816763c2c14aa7338c7fca3182566de1cad3ac"
Oct 29 22:16:33.012165 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:33.012126    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4"]
Oct 29 22:16:33.031020 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:33.030979    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a9e5f-l46x4"]
Oct 29 22:16:33.056926 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:33.056892    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4067333-0059-4b1f-9f15-974985dba84a" path="/var/lib/kubelet/pods/c4067333-0059-4b1f-9f15-974985dba84a/volumes"
Oct 29 22:16:33.210540 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:33.210505    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr" podUID="d37a81f7-dff2-429d-8dd1-c5d626299df8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:16:33.210692 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:33.210582    4962 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd" podUID="cee4891f-98dd-4ea8-a470-0f85ea62b849" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 22:16:39.719525 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:16:39.719491    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:16:40.035932 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.035898    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr"
Oct 29 22:16:40.069202 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.069174    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d37a81f7-dff2-429d-8dd1-c5d626299df8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37a81f7-dff2-429d-8dd1-c5d626299df8-plugins-config-volume\") pod \"d37a81f7-dff2-429d-8dd1-c5d626299df8\" (UID: \"d37a81f7-dff2-429d-8dd1-c5d626299df8\") "
Oct 29 22:16:40.069390 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.069225    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d37a81f7-dff2-429d-8dd1-c5d626299df8-kube-api-access-zshv9\" (UniqueName: \"kubernetes.io/projected/d37a81f7-dff2-429d-8dd1-c5d626299df8-kube-api-access-zshv9\") pod \"d37a81f7-dff2-429d-8dd1-c5d626299df8\" (UID: \"d37a81f7-dff2-429d-8dd1-c5d626299df8\") "
Oct 29 22:16:40.070185 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.070156    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d37a81f7-dff2-429d-8dd1-c5d626299df8-plugins-config-volume" pod "d37a81f7-dff2-429d-8dd1-c5d626299df8" (UID: "d37a81f7-dff2-429d-8dd1-c5d626299df8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:40.079181 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.078926    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37a81f7-dff2-429d-8dd1-c5d626299df8-kube-api-access-zshv9" pod "d37a81f7-dff2-429d-8dd1-c5d626299df8" (UID: "d37a81f7-dff2-429d-8dd1-c5d626299df8"). InnerVolumeSpecName "kube-api-access-zshv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:40.086013 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.083761    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd"
Oct 29 22:16:40.110988 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.110947    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"]
Oct 29 22:16:40.111215 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.111192    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt" podUID="730edf47-8b07-477a-acfc-47faaaacfc84" containerName="istio-proxy" containerID="cri-o://8f72340e66bc196b63adb77b7aeb06119281c2d3ecb65558ef9a6a315f89fbe4" gracePeriod=30
Oct 29 22:16:40.114681 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.114643    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"]
Oct 29 22:16:40.115030 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.114859    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz" podUID="8054bc19-6db7-462f-8af0-335903ce62f3" containerName="istio-proxy" containerID="cri-o://c69b96ab47e84cbe96c8a6c22e84043a9517fba53d146ed6885fbeb0e930af70" gracePeriod=30
Oct 29 22:16:40.116373 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.116352    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"]
Oct 29 22:16:40.116542 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.116523    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z" podUID="132ed775-c9e6-41f1-b224-fff550c6e073" containerName="istio-proxy" containerID="cri-o://4fb526ee75ea34c842db621e76967cacd7903f6db2237b901b7372b19a60661b" gracePeriod=30
Oct 29 22:16:40.123529 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.123183    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"]
Oct 29 22:16:40.123529 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.123402    4962 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr" podUID="d91917d2-1f66-4ee4-a312-57398651739e" containerName="istio-proxy" containerID="cri-o://bc30e57de07692cb4a3806e7b3e8b9681bbed19d93e483b2fc26d80bd7f5d165" gracePeriod=30
Oct 29 22:16:40.170485 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.170451    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee4891f-98dd-4ea8-a470-0f85ea62b849-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee4891f-98dd-4ea8-a470-0f85ea62b849-plugins-config-volume\") pod \"cee4891f-98dd-4ea8-a470-0f85ea62b849\" (UID: \"cee4891f-98dd-4ea8-a470-0f85ea62b849\") "
Oct 29 22:16:40.170653 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.170505    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee4891f-98dd-4ea8-a470-0f85ea62b849-kube-api-access-4bsbz\" (UniqueName: \"kubernetes.io/projected/cee4891f-98dd-4ea8-a470-0f85ea62b849-kube-api-access-4bsbz\") pod \"cee4891f-98dd-4ea8-a470-0f85ea62b849\" (UID: \"cee4891f-98dd-4ea8-a470-0f85ea62b849\") "
Oct 29 22:16:40.170821 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.170799    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee4891f-98dd-4ea8-a470-0f85ea62b849-plugins-config-volume" pod "cee4891f-98dd-4ea8-a470-0f85ea62b849" (UID: "cee4891f-98dd-4ea8-a470-0f85ea62b849"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:40.170898 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.170885    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cee4891f-98dd-4ea8-a470-0f85ea62b849-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:40.170940 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.170901    4962 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37a81f7-dff2-429d-8dd1-c5d626299df8-plugins-config-volume\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:40.170940 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.170911    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zshv9\" (UniqueName: \"kubernetes.io/projected/d37a81f7-dff2-429d-8dd1-c5d626299df8-kube-api-access-zshv9\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:40.172810 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.172782    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee4891f-98dd-4ea8-a470-0f85ea62b849-kube-api-access-4bsbz" pod "cee4891f-98dd-4ea8-a470-0f85ea62b849" (UID: "cee4891f-98dd-4ea8-a470-0f85ea62b849"). InnerVolumeSpecName "kube-api-access-4bsbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:40.272014 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.271928    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bsbz\" (UniqueName: \"kubernetes.io/projected/cee4891f-98dd-4ea8-a470-0f85ea62b849-kube-api-access-4bsbz\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:40.414960 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.414925    4962 generic.go:356] "Generic (PLEG): container finished" podID="d37a81f7-dff2-429d-8dd1-c5d626299df8" containerID="4092a0b64316186a01c4436afa0756c378079bc99d5911cd76886087054c1c2d" exitCode=0
Oct 29 22:16:40.415163 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.414969    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr"
Oct 29 22:16:40.415163 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.415010    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr" event={"ID":"d37a81f7-dff2-429d-8dd1-c5d626299df8","Type":"ContainerDied","Data":"4092a0b64316186a01c4436afa0756c378079bc99d5911cd76886087054c1c2d"}
Oct 29 22:16:40.415163 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.415061    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr" event={"ID":"d37a81f7-dff2-429d-8dd1-c5d626299df8","Type":"ContainerDied","Data":"fc68b1d9164dbdaa5eaa70a299266cd4c2a1e48ceeb12124ecfd82620fda616c"}
Oct 29 22:16:40.415163 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.415083    4962 scope.go:122] "RemoveContainer" containerID="4092a0b64316186a01c4436afa0756c378079bc99d5911cd76886087054c1c2d"
Oct 29 22:16:40.416471 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.416450    4962 generic.go:356] "Generic (PLEG): container finished" podID="cee4891f-98dd-4ea8-a470-0f85ea62b849" containerID="d541e0eb15d76e65530d3b4835498b30cd423ad44a84e9c1cfd1d1f2e017be85" exitCode=0
Oct 29 22:16:40.416558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.416485    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd"
Oct 29 22:16:40.416558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.416518    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd" event={"ID":"cee4891f-98dd-4ea8-a470-0f85ea62b849","Type":"ContainerDied","Data":"d541e0eb15d76e65530d3b4835498b30cd423ad44a84e9c1cfd1d1f2e017be85"}
Oct 29 22:16:40.416558 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.416556    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd" event={"ID":"cee4891f-98dd-4ea8-a470-0f85ea62b849","Type":"ContainerDied","Data":"4a072a9f889cd09df3f647ed40dfba29e8ba4df4f5b771dc7962a105bfac1b30"}
Oct 29 22:16:40.424879 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.424742    4962 scope.go:122] "RemoveContainer" containerID="4092a0b64316186a01c4436afa0756c378079bc99d5911cd76886087054c1c2d"
Oct 29 22:16:40.425118 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:16:40.425075    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4092a0b64316186a01c4436afa0756c378079bc99d5911cd76886087054c1c2d\": container with ID starting with 4092a0b64316186a01c4436afa0756c378079bc99d5911cd76886087054c1c2d not found: ID does not exist" containerID="4092a0b64316186a01c4436afa0756c378079bc99d5911cd76886087054c1c2d"
Oct 29 22:16:40.425240 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.425131    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4092a0b64316186a01c4436afa0756c378079bc99d5911cd76886087054c1c2d"} err="failed to get container status \"4092a0b64316186a01c4436afa0756c378079bc99d5911cd76886087054c1c2d\": rpc error: code = NotFound desc = could not find container \"4092a0b64316186a01c4436afa0756c378079bc99d5911cd76886087054c1c2d\": container with ID starting with 4092a0b64316186a01c4436afa0756c378079bc99d5911cd76886087054c1c2d not found: ID does not exist"
Oct 29 22:16:40.425240 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.425157    4962 scope.go:122] "RemoveContainer" containerID="d541e0eb15d76e65530d3b4835498b30cd423ad44a84e9c1cfd1d1f2e017be85"
Oct 29 22:16:40.433434 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.433418    4962 scope.go:122] "RemoveContainer" containerID="d541e0eb15d76e65530d3b4835498b30cd423ad44a84e9c1cfd1d1f2e017be85"
Oct 29 22:16:40.433667 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:16:40.433646    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d541e0eb15d76e65530d3b4835498b30cd423ad44a84e9c1cfd1d1f2e017be85\": container with ID starting with d541e0eb15d76e65530d3b4835498b30cd423ad44a84e9c1cfd1d1f2e017be85 not found: ID does not exist" containerID="d541e0eb15d76e65530d3b4835498b30cd423ad44a84e9c1cfd1d1f2e017be85"
Oct 29 22:16:40.433730 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.433677    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d541e0eb15d76e65530d3b4835498b30cd423ad44a84e9c1cfd1d1f2e017be85"} err="failed to get container status \"d541e0eb15d76e65530d3b4835498b30cd423ad44a84e9c1cfd1d1f2e017be85\": rpc error: code = NotFound desc = could not find container \"d541e0eb15d76e65530d3b4835498b30cd423ad44a84e9c1cfd1d1f2e017be85\": container with ID starting with d541e0eb15d76e65530d3b4835498b30cd423ad44a84e9c1cfd1d1f2e017be85 not found: ID does not exist"
Oct 29 22:16:40.463779 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.463754    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd"]
Oct 29 22:16:40.467716 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.467695    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-8a9e5f-df657cd95-8nqkd"]
Oct 29 22:16:40.499574 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.499549    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr"]
Oct 29 22:16:40.503320 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:40.503280    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8a9e5f-5979656864-vjgwr"]
Oct 29 22:16:41.056175 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:41.056102    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cee4891f-98dd-4ea8-a470-0f85ea62b849" path="/var/lib/kubelet/pods/cee4891f-98dd-4ea8-a470-0f85ea62b849/volumes"
Oct 29 22:16:41.056556 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:41.056484    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d37a81f7-dff2-429d-8dd1-c5d626299df8" path="/var/lib/kubelet/pods/d37a81f7-dff2-429d-8dd1-c5d626299df8/volumes"
Oct 29 22:16:42.233791 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:42.233764    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-7bbc6bb9d7-92hd8_ac207213-067e-4673-b4b2-630469c878a8/kuadrant-console-plugin/0.log"
Oct 29 22:16:43.218778 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:43.218750    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-68669df9c4-ktqlt_3391c781-0e29-49eb-a190-20229a0ab066/manager/0.log"
Oct 29 22:16:44.394328 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:44.394278    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-767df45bdd-6djqs_68e70e0c-c896-4641-b5d7-f97e8bf2ce79/manager/0.log"
Oct 29 22:16:45.414638 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.414588    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 22:16:45.435061 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.435030    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 22:16:45.445685 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.445633    4962 generic.go:356] "Generic (PLEG): container finished" podID="132ed775-c9e6-41f1-b224-fff550c6e073" containerID="4fb526ee75ea34c842db621e76967cacd7903f6db2237b901b7372b19a60661b" exitCode=0
Oct 29 22:16:45.445839 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.445683    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"
Oct 29 22:16:45.445839 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.445733    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z" event={"ID":"132ed775-c9e6-41f1-b224-fff550c6e073","Type":"ContainerDied","Data":"4fb526ee75ea34c842db621e76967cacd7903f6db2237b901b7372b19a60661b"}
Oct 29 22:16:45.445839 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.445782    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z" event={"ID":"132ed775-c9e6-41f1-b224-fff550c6e073","Type":"ContainerDied","Data":"89cbff96b976d2351e3a899b258d67827b48d64c002a283986e3a7b9992e3169"}
Oct 29 22:16:45.445839 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.445811    4962 scope.go:122] "RemoveContainer" containerID="4fb526ee75ea34c842db621e76967cacd7903f6db2237b901b7372b19a60661b"
Oct 29 22:16:45.447511 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.447492    4962 generic.go:356] "Generic (PLEG): container finished" podID="d91917d2-1f66-4ee4-a312-57398651739e" containerID="bc30e57de07692cb4a3806e7b3e8b9681bbed19d93e483b2fc26d80bd7f5d165" exitCode=0
Oct 29 22:16:45.447704 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.447688    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"
Oct 29 22:16:45.447845 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.447828    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr" event={"ID":"d91917d2-1f66-4ee4-a312-57398651739e","Type":"ContainerDied","Data":"bc30e57de07692cb4a3806e7b3e8b9681bbed19d93e483b2fc26d80bd7f5d165"}
Oct 29 22:16:45.447918 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.447873    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr" event={"ID":"d91917d2-1f66-4ee4-a312-57398651739e","Type":"ContainerDied","Data":"f680282948b430472f66ef341e19df3efca6c5e30a97a1ea9a815516120a5443"}
Oct 29 22:16:45.450018 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.449871    4962 generic.go:356] "Generic (PLEG): container finished" podID="8054bc19-6db7-462f-8af0-335903ce62f3" containerID="c69b96ab47e84cbe96c8a6c22e84043a9517fba53d146ed6885fbeb0e930af70" exitCode=0
Oct 29 22:16:45.450018 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.449943    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz" event={"ID":"8054bc19-6db7-462f-8af0-335903ce62f3","Type":"ContainerDied","Data":"c69b96ab47e84cbe96c8a6c22e84043a9517fba53d146ed6885fbeb0e930af70"}
Oct 29 22:16:45.452025 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.451996    4962 generic.go:356] "Generic (PLEG): container finished" podID="730edf47-8b07-477a-acfc-47faaaacfc84" containerID="8f72340e66bc196b63adb77b7aeb06119281c2d3ecb65558ef9a6a315f89fbe4" exitCode=0
Oct 29 22:16:45.452129 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.452045    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt" event={"ID":"730edf47-8b07-477a-acfc-47faaaacfc84","Type":"ContainerDied","Data":"8f72340e66bc196b63adb77b7aeb06119281c2d3ecb65558ef9a6a315f89fbe4"}
Oct 29 22:16:45.458178 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.458148    4962 scope.go:122] "RemoveContainer" containerID="4fb526ee75ea34c842db621e76967cacd7903f6db2237b901b7372b19a60661b"
Oct 29 22:16:45.458487 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:16:45.458468    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fb526ee75ea34c842db621e76967cacd7903f6db2237b901b7372b19a60661b\": container with ID starting with 4fb526ee75ea34c842db621e76967cacd7903f6db2237b901b7372b19a60661b not found: ID does not exist" containerID="4fb526ee75ea34c842db621e76967cacd7903f6db2237b901b7372b19a60661b"
Oct 29 22:16:45.458582 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.458494    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fb526ee75ea34c842db621e76967cacd7903f6db2237b901b7372b19a60661b"} err="failed to get container status \"4fb526ee75ea34c842db621e76967cacd7903f6db2237b901b7372b19a60661b\": rpc error: code = NotFound desc = could not find container \"4fb526ee75ea34c842db621e76967cacd7903f6db2237b901b7372b19a60661b\": container with ID starting with 4fb526ee75ea34c842db621e76967cacd7903f6db2237b901b7372b19a60661b not found: ID does not exist"
Oct 29 22:16:45.458582 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.458512    4962 scope.go:122] "RemoveContainer" containerID="bc30e57de07692cb4a3806e7b3e8b9681bbed19d93e483b2fc26d80bd7f5d165"
Oct 29 22:16:45.469395 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.469371    4962 scope.go:122] "RemoveContainer" containerID="bc30e57de07692cb4a3806e7b3e8b9681bbed19d93e483b2fc26d80bd7f5d165"
Oct 29 22:16:45.469740 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:16:45.469718    4962 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc30e57de07692cb4a3806e7b3e8b9681bbed19d93e483b2fc26d80bd7f5d165\": container with ID starting with bc30e57de07692cb4a3806e7b3e8b9681bbed19d93e483b2fc26d80bd7f5d165 not found: ID does not exist" containerID="bc30e57de07692cb4a3806e7b3e8b9681bbed19d93e483b2fc26d80bd7f5d165"
Oct 29 22:16:45.469829 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.469760    4962 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc30e57de07692cb4a3806e7b3e8b9681bbed19d93e483b2fc26d80bd7f5d165"} err="failed to get container status \"bc30e57de07692cb4a3806e7b3e8b9681bbed19d93e483b2fc26d80bd7f5d165\": rpc error: code = NotFound desc = could not find container \"bc30e57de07692cb4a3806e7b3e8b9681bbed19d93e483b2fc26d80bd7f5d165\": container with ID starting with bc30e57de07692cb4a3806e7b3e8b9681bbed19d93e483b2fc26d80bd7f5d165 not found: ID does not exist"
Oct 29 22:16:45.481665 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.481644    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 22:16:45.483647 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.483630    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 22:16:45.534071 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534043    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-workload-certs\") pod \"132ed775-c9e6-41f1-b224-fff550c6e073\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") "
Oct 29 22:16:45.534229 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534078    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-credential-socket\") pod \"730edf47-8b07-477a-acfc-47faaaacfc84\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") "
Oct 29 22:16:45.534229 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534106    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/730edf47-8b07-477a-acfc-47faaaacfc84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/730edf47-8b07-477a-acfc-47faaaacfc84-istio-podinfo\") pod \"730edf47-8b07-477a-acfc-47faaaacfc84\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") "
Oct 29 22:16:45.534229 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534143    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-istio-data\") pod \"132ed775-c9e6-41f1-b224-fff550c6e073\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") "
Oct 29 22:16:45.534229 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534169    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-workload-socket\") pod \"730edf47-8b07-477a-acfc-47faaaacfc84\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") "
Oct 29 22:16:45.534229 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534205    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/132ed775-c9e6-41f1-b224-fff550c6e073-kube-api-access-6xdgm\" (UniqueName: \"kubernetes.io/projected/132ed775-c9e6-41f1-b224-fff550c6e073-kube-api-access-6xdgm\") pod \"132ed775-c9e6-41f1-b224-fff550c6e073\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") "
Oct 29 22:16:45.534608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534235    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-workload-certs\") pod \"730edf47-8b07-477a-acfc-47faaaacfc84\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") "
Oct 29 22:16:45.534608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534261    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8054bc19-6db7-462f-8af0-335903ce62f3-istio-token\" (UniqueName: \"kubernetes.io/projected/8054bc19-6db7-462f-8af0-335903ce62f3-istio-token\") pod \"8054bc19-6db7-462f-8af0-335903ce62f3\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") "
Oct 29 22:16:45.534608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534303    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730edf47-8b07-477a-acfc-47faaaacfc84-istio-token\" (UniqueName: \"kubernetes.io/projected/730edf47-8b07-477a-acfc-47faaaacfc84-istio-token\") pod \"730edf47-8b07-477a-acfc-47faaaacfc84\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") "
Oct 29 22:16:45.534608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534337    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-credential-socket\") pod \"d91917d2-1f66-4ee4-a312-57398651739e\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") "
Oct 29 22:16:45.534608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534362    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8054bc19-6db7-462f-8af0-335903ce62f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8054bc19-6db7-462f-8af0-335903ce62f3-istiod-ca-cert\") pod \"8054bc19-6db7-462f-8af0-335903ce62f3\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") "
Oct 29 22:16:45.534608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534387    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-credential-socket\") pod \"132ed775-c9e6-41f1-b224-fff550c6e073\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") "
Oct 29 22:16:45.534608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534414    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-istio-envoy\") pod \"132ed775-c9e6-41f1-b224-fff550c6e073\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") "
Oct 29 22:16:45.534608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534444    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-workload-socket\") pod \"8054bc19-6db7-462f-8af0-335903ce62f3\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") "
Oct 29 22:16:45.534608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534473    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-istio-data\") pod \"730edf47-8b07-477a-acfc-47faaaacfc84\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") "
Oct 29 22:16:45.534608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534516    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91917d2-1f66-4ee4-a312-57398651739e-kube-api-access-vzqfj\" (UniqueName: \"kubernetes.io/projected/d91917d2-1f66-4ee4-a312-57398651739e-kube-api-access-vzqfj\") pod \"d91917d2-1f66-4ee4-a312-57398651739e\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") "
Oct 29 22:16:45.534608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534541    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-istio-envoy\") pod \"730edf47-8b07-477a-acfc-47faaaacfc84\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") "
Oct 29 22:16:45.534608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534567    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d91917d2-1f66-4ee4-a312-57398651739e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91917d2-1f66-4ee4-a312-57398651739e-istio-podinfo\") pod \"d91917d2-1f66-4ee4-a312-57398651739e\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") "
Oct 29 22:16:45.534608 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534594    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-istio-data\") pod \"d91917d2-1f66-4ee4-a312-57398651739e\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") "
Oct 29 22:16:45.535336 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534631    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/132ed775-c9e6-41f1-b224-fff550c6e073-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132ed775-c9e6-41f1-b224-fff550c6e073-istio-podinfo\") pod \"132ed775-c9e6-41f1-b224-fff550c6e073\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") "
Oct 29 22:16:45.535336 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534650    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-istio-envoy\") pod \"8054bc19-6db7-462f-8af0-335903ce62f3\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") "
Oct 29 22:16:45.535336 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534667    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d91917d2-1f66-4ee4-a312-57398651739e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91917d2-1f66-4ee4-a312-57398651739e-istiod-ca-cert\") pod \"d91917d2-1f66-4ee4-a312-57398651739e\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") "
Oct 29 22:16:45.535336 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534690    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-workload-certs\") pod \"8054bc19-6db7-462f-8af0-335903ce62f3\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") "
Oct 29 22:16:45.535336 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534725    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/132ed775-c9e6-41f1-b224-fff550c6e073-istio-token\" (UniqueName: \"kubernetes.io/projected/132ed775-c9e6-41f1-b224-fff550c6e073-istio-token\") pod \"132ed775-c9e6-41f1-b224-fff550c6e073\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") "
Oct 29 22:16:45.535336 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534743    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-istio-envoy\") pod \"d91917d2-1f66-4ee4-a312-57398651739e\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") "
Oct 29 22:16:45.535336 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534759    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8054bc19-6db7-462f-8af0-335903ce62f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8054bc19-6db7-462f-8af0-335903ce62f3-istio-podinfo\") pod \"8054bc19-6db7-462f-8af0-335903ce62f3\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") "
Oct 29 22:16:45.535336 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534796    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-istio-data\") pod \"8054bc19-6db7-462f-8af0-335903ce62f3\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") "
Oct 29 22:16:45.535336 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534833    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/132ed775-c9e6-41f1-b224-fff550c6e073-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132ed775-c9e6-41f1-b224-fff550c6e073-istiod-ca-cert\") pod \"132ed775-c9e6-41f1-b224-fff550c6e073\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") "
Oct 29 22:16:45.535336 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534856    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-credential-socket\") pod \"8054bc19-6db7-462f-8af0-335903ce62f3\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") "
Oct 29 22:16:45.535885 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.535536    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-workload-socket\") pod \"d91917d2-1f66-4ee4-a312-57398651739e\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") "
Oct 29 22:16:45.535885 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.535599    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d91917d2-1f66-4ee4-a312-57398651739e-istio-token\" (UniqueName: \"kubernetes.io/projected/d91917d2-1f66-4ee4-a312-57398651739e-istio-token\") pod \"d91917d2-1f66-4ee4-a312-57398651739e\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") "
Oct 29 22:16:45.535885 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.535636    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-workload-socket\") pod \"132ed775-c9e6-41f1-b224-fff550c6e073\" (UID: \"132ed775-c9e6-41f1-b224-fff550c6e073\") "
Oct 29 22:16:45.535885 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.535691    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-workload-certs\") pod \"d91917d2-1f66-4ee4-a312-57398651739e\" (UID: \"d91917d2-1f66-4ee4-a312-57398651739e\") "
Oct 29 22:16:45.536578 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534411    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-workload-certs" pod "132ed775-c9e6-41f1-b224-fff550c6e073" (UID: "132ed775-c9e6-41f1-b224-fff550c6e073"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.536578 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534622    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-workload-certs" pod "730edf47-8b07-477a-acfc-47faaaacfc84" (UID: "730edf47-8b07-477a-acfc-47faaaacfc84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.536578 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534798    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-credential-socket" pod "132ed775-c9e6-41f1-b224-fff550c6e073" (UID: "132ed775-c9e6-41f1-b224-fff550c6e073"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.536578 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.534849    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-credential-socket" pod "730edf47-8b07-477a-acfc-47faaaacfc84" (UID: "730edf47-8b07-477a-acfc-47faaaacfc84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.536578 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.535068    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-credential-socket" pod "8054bc19-6db7-462f-8af0-335903ce62f3" (UID: "8054bc19-6db7-462f-8af0-335903ce62f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.536578 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.535505    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-workload-socket" pod "730edf47-8b07-477a-acfc-47faaaacfc84" (UID: "730edf47-8b07-477a-acfc-47faaaacfc84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.536578 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.535679    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-istio-data" pod "132ed775-c9e6-41f1-b224-fff550c6e073" (UID: "132ed775-c9e6-41f1-b224-fff550c6e073"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.536578 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.535715    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8054bc19-6db7-462f-8af0-335903ce62f3-istiod-ca-cert" pod "8054bc19-6db7-462f-8af0-335903ce62f3" (UID: "8054bc19-6db7-462f-8af0-335903ce62f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:45.536578 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.535893    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-workload-socket" pod "d91917d2-1f66-4ee4-a312-57398651739e" (UID: "d91917d2-1f66-4ee4-a312-57398651739e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.536578 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.536071    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-workload-socket" pod "8054bc19-6db7-462f-8af0-335903ce62f3" (UID: "8054bc19-6db7-462f-8af0-335903ce62f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.536578 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.536308    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-credential-socket" pod "d91917d2-1f66-4ee4-a312-57398651739e" (UID: "d91917d2-1f66-4ee4-a312-57398651739e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.536578 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.536330    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-istio-data" pod "730edf47-8b07-477a-acfc-47faaaacfc84" (UID: "730edf47-8b07-477a-acfc-47faaaacfc84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.539014 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.536896    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/730edf47-8b07-477a-acfc-47faaaacfc84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/730edf47-8b07-477a-acfc-47faaaacfc84-istiod-ca-cert\") pod \"730edf47-8b07-477a-acfc-47faaaacfc84\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") "
Oct 29 22:16:45.539014 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.536977    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730edf47-8b07-477a-acfc-47faaaacfc84-kube-api-access-4sk4s\" (UniqueName: \"kubernetes.io/projected/730edf47-8b07-477a-acfc-47faaaacfc84-kube-api-access-4sk4s\") pod \"730edf47-8b07-477a-acfc-47faaaacfc84\" (UID: \"730edf47-8b07-477a-acfc-47faaaacfc84\") "
Oct 29 22:16:45.539014 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.537015    4962 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8054bc19-6db7-462f-8af0-335903ce62f3-kube-api-access-6x7vt\" (UniqueName: \"kubernetes.io/projected/8054bc19-6db7-462f-8af0-335903ce62f3-kube-api-access-6x7vt\") pod \"8054bc19-6db7-462f-8af0-335903ce62f3\" (UID: \"8054bc19-6db7-462f-8af0-335903ce62f3\") "
Oct 29 22:16:45.539014 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.538573    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-istio-data" pod "d91917d2-1f66-4ee4-a312-57398651739e" (UID: "d91917d2-1f66-4ee4-a312-57398651739e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.539387 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.539348    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d91917d2-1f66-4ee4-a312-57398651739e-istiod-ca-cert" pod "d91917d2-1f66-4ee4-a312-57398651739e" (UID: "d91917d2-1f66-4ee4-a312-57398651739e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:45.539733 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.539621    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-workload-certs" pod "8054bc19-6db7-462f-8af0-335903ce62f3" (UID: "8054bc19-6db7-462f-8af0-335903ce62f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.539733 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.539661    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730edf47-8b07-477a-acfc-47faaaacfc84-istiod-ca-cert" pod "730edf47-8b07-477a-acfc-47faaaacfc84" (UID: "730edf47-8b07-477a-acfc-47faaaacfc84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:45.539935 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.539917    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-workload-socket" pod "132ed775-c9e6-41f1-b224-fff550c6e073" (UID: "132ed775-c9e6-41f1-b224-fff550c6e073"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.540336 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.540112    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-workload-certs" pod "d91917d2-1f66-4ee4-a312-57398651739e" (UID: "d91917d2-1f66-4ee4-a312-57398651739e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.541674 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.541100    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-istio-data" pod "8054bc19-6db7-462f-8af0-335903ce62f3" (UID: "8054bc19-6db7-462f-8af0-335903ce62f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.541674 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.541496    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/132ed775-c9e6-41f1-b224-fff550c6e073-istiod-ca-cert" pod "132ed775-c9e6-41f1-b224-fff550c6e073" (UID: "132ed775-c9e6-41f1-b224-fff550c6e073"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:16:45.541674 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.541581    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8054bc19-6db7-462f-8af0-335903ce62f3-istio-token" pod "8054bc19-6db7-462f-8af0-335903ce62f3" (UID: "8054bc19-6db7-462f-8af0-335903ce62f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:45.541674 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.541635    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/730edf47-8b07-477a-acfc-47faaaacfc84-istio-podinfo" pod "730edf47-8b07-477a-acfc-47faaaacfc84" (UID: "730edf47-8b07-477a-acfc-47faaaacfc84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:16:45.542904 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.542869    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730edf47-8b07-477a-acfc-47faaaacfc84-istio-token" pod "730edf47-8b07-477a-acfc-47faaaacfc84" (UID: "730edf47-8b07-477a-acfc-47faaaacfc84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:45.543498 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.543471    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8054bc19-6db7-462f-8af0-335903ce62f3-istio-podinfo" pod "8054bc19-6db7-462f-8af0-335903ce62f3" (UID: "8054bc19-6db7-462f-8af0-335903ce62f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:16:45.544805 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.544163    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/132ed775-c9e6-41f1-b224-fff550c6e073-istio-token" pod "132ed775-c9e6-41f1-b224-fff550c6e073" (UID: "132ed775-c9e6-41f1-b224-fff550c6e073"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:45.544805 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.544447    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-istio-envoy" pod "d91917d2-1f66-4ee4-a312-57398651739e" (UID: "d91917d2-1f66-4ee4-a312-57398651739e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.544995 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.544879    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8054bc19-6db7-462f-8af0-335903ce62f3-kube-api-access-6x7vt" pod "8054bc19-6db7-462f-8af0-335903ce62f3" (UID: "8054bc19-6db7-462f-8af0-335903ce62f3"). InnerVolumeSpecName "kube-api-access-6x7vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:45.545998 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.545975    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/132ed775-c9e6-41f1-b224-fff550c6e073-istio-podinfo" pod "132ed775-c9e6-41f1-b224-fff550c6e073" (UID: "132ed775-c9e6-41f1-b224-fff550c6e073"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:16:45.545998 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.545991    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/132ed775-c9e6-41f1-b224-fff550c6e073-kube-api-access-6xdgm" pod "132ed775-c9e6-41f1-b224-fff550c6e073" (UID: "132ed775-c9e6-41f1-b224-fff550c6e073"). InnerVolumeSpecName "kube-api-access-6xdgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:45.546178 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.546066    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-istio-envoy" pod "132ed775-c9e6-41f1-b224-fff550c6e073" (UID: "132ed775-c9e6-41f1-b224-fff550c6e073"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.546246 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.546183    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730edf47-8b07-477a-acfc-47faaaacfc84-kube-api-access-4sk4s" pod "730edf47-8b07-477a-acfc-47faaaacfc84" (UID: "730edf47-8b07-477a-acfc-47faaaacfc84"). InnerVolumeSpecName "kube-api-access-4sk4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:45.546246 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.546207    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-istio-envoy" pod "730edf47-8b07-477a-acfc-47faaaacfc84" (UID: "730edf47-8b07-477a-acfc-47faaaacfc84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.546384 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.546278    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-istio-envoy" pod "8054bc19-6db7-462f-8af0-335903ce62f3" (UID: "8054bc19-6db7-462f-8af0-335903ce62f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:16:45.546384 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.546325    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91917d2-1f66-4ee4-a312-57398651739e-kube-api-access-vzqfj" pod "d91917d2-1f66-4ee4-a312-57398651739e" (UID: "d91917d2-1f66-4ee4-a312-57398651739e"). InnerVolumeSpecName "kube-api-access-vzqfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:45.546484 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.546401    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d91917d2-1f66-4ee4-a312-57398651739e-istio-podinfo" pod "d91917d2-1f66-4ee4-a312-57398651739e" (UID: "d91917d2-1f66-4ee4-a312-57398651739e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:16:45.547754 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.547736    4962 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91917d2-1f66-4ee4-a312-57398651739e-istio-token" pod "d91917d2-1f66-4ee4-a312-57398651739e" (UID: "d91917d2-1f66-4ee4-a312-57398651739e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:16:45.638924 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.638890    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.638924 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.638913    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/132ed775-c9e6-41f1-b224-fff550c6e073-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.638924 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.638926    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.638938    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.638947    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91917d2-1f66-4ee4-a312-57398651739e-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.638955    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.638963    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.638971    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/730edf47-8b07-477a-acfc-47faaaacfc84-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.638979    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sk4s\" (UniqueName: \"kubernetes.io/projected/730edf47-8b07-477a-acfc-47faaaacfc84-kube-api-access-4sk4s\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.638987    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x7vt\" (UniqueName: \"kubernetes.io/projected/8054bc19-6db7-462f-8af0-335903ce62f3-kube-api-access-6x7vt\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.638997    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639011    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639019    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/730edf47-8b07-477a-acfc-47faaaacfc84-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639028    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639047    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639057    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xdgm\" (UniqueName: \"kubernetes.io/projected/132ed775-c9e6-41f1-b224-fff550c6e073-kube-api-access-6xdgm\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639066    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639075    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8054bc19-6db7-462f-8af0-335903ce62f3-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639086    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/730edf47-8b07-477a-acfc-47faaaacfc84-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639094    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639101    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8054bc19-6db7-462f-8af0-335903ce62f3-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639109    4962 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-credential-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639118    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/132ed775-c9e6-41f1-b224-fff550c6e073-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639126    4962 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-workload-socket\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639133    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639142    4962 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzqfj\" (UniqueName: \"kubernetes.io/projected/d91917d2-1f66-4ee4-a312-57398651739e-kube-api-access-vzqfj\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639151    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/730edf47-8b07-477a-acfc-47faaaacfc84-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639164    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91917d2-1f66-4ee4-a312-57398651739e-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.639160 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639172    4962 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-istio-data\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.640012 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639181    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/132ed775-c9e6-41f1-b224-fff550c6e073-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.640012 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639188    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.640012 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639196    4962 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91917d2-1f66-4ee4-a312-57398651739e-istiod-ca-cert\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.640012 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639204    4962 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8054bc19-6db7-462f-8af0-335903ce62f3-workload-certs\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.640012 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639212    4962 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/132ed775-c9e6-41f1-b224-fff550c6e073-istio-token\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.640012 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639219    4962 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91917d2-1f66-4ee4-a312-57398651739e-istio-envoy\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.640012 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.639227    4962 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8054bc19-6db7-462f-8af0-335903ce62f3-istio-podinfo\") on node \"ip-10-0-129-146.ec2.internal\" DevicePath \"\""
Oct 29 22:16:45.791739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.791709    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"]
Oct 29 22:16:45.796068 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.796044    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-87s9z"]
Oct 29 22:16:45.827408 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.827384    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"]
Oct 29 22:16:45.831587 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:45.831566    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-fqjcr"]
Oct 29 22:16:46.457925 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:46.457855    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz" event={"ID":"8054bc19-6db7-462f-8af0-335903ce62f3","Type":"ContainerDied","Data":"7b98fe2bd31c7f17e7b3af8c88838adf091090cda4bde2fe4c3403e7c2252f09"}
Oct 29 22:16:46.457925 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:46.457866    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"
Oct 29 22:16:46.457925 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:46.457909    4962 scope.go:122] "RemoveContainer" containerID="c69b96ab47e84cbe96c8a6c22e84043a9517fba53d146ed6885fbeb0e930af70"
Oct 29 22:16:46.463758 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:46.463736    4962 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"
Oct 29 22:16:46.463891 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:46.463733    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt" event={"ID":"730edf47-8b07-477a-acfc-47faaaacfc84","Type":"ContainerDied","Data":"0d5703ca13a73adaef8685901e22556a82f0d7e2a46d561aa403c201a00b6c06"}
Oct 29 22:16:46.469403 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:46.469386    4962 scope.go:122] "RemoveContainer" containerID="8f72340e66bc196b63adb77b7aeb06119281c2d3ecb65558ef9a6a315f89fbe4"
Oct 29 22:16:46.522257 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:46.522232    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"]
Oct 29 22:16:46.534122 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:46.534094    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-9knmz"]
Oct 29 22:16:46.561360 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:46.561339    4962 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"]
Oct 29 22:16:46.565677 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:46.565656    4962 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a9e5f-openshift-default-5c996d895f-7f2vt"]
Oct 29 22:16:47.056263 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:47.056229    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="132ed775-c9e6-41f1-b224-fff550c6e073" path="/var/lib/kubelet/pods/132ed775-c9e6-41f1-b224-fff550c6e073/volumes"
Oct 29 22:16:47.056683 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:47.056669    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="730edf47-8b07-477a-acfc-47faaaacfc84" path="/var/lib/kubelet/pods/730edf47-8b07-477a-acfc-47faaaacfc84/volumes"
Oct 29 22:16:47.057074 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:47.057060    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8054bc19-6db7-462f-8af0-335903ce62f3" path="/var/lib/kubelet/pods/8054bc19-6db7-462f-8af0-335903ce62f3/volumes"
Oct 29 22:16:47.057726 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:47.057706    4962 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d91917d2-1f66-4ee4-a312-57398651739e" path="/var/lib/kubelet/pods/d91917d2-1f66-4ee4-a312-57398651739e/volumes"
Oct 29 22:16:57.198036 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:16:57.198000    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-767df45bdd-6djqs_68e70e0c-c896-4641-b5d7-f97e8bf2ce79/manager/0.log"
Oct 29 22:17:01.567735 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:01.567710    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9fe3d91f-2z5j9_adbf3c9f-a2e2-49b9-bef2-a9a0847b0374/revoke-keys/0.log"
Oct 29 22:17:01.766226 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:01.766202    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e74b984d-q7m8g_aebb8907-436e-460b-a5a5-9fc278a2ada4/revoke-keys/0.log"
Oct 29 22:17:04.348060 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:04.347979    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-7bbc6bb9d7-92hd8_ac207213-067e-4673-b4b2-630469c878a8/kuadrant-console-plugin/0.log"
Oct 29 22:17:05.344132 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:05.344102    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-68669df9c4-ktqlt_3391c781-0e29-49eb-a190-20229a0ab066/manager/0.log"
Oct 29 22:17:06.744720 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:06.744686    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-bbmrg_2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5/discovery/0.log"
Oct 29 22:17:08.323989 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:08.323960    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-j6f7m_a92fb3ec-6127-46dd-a906-e84aec50ccae/istio-proxy/0.log"
Oct 29 22:17:08.526222 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:08.526198    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-krsk7_aacf5dff-ff07-4c5c-a98f-db3d55beb6b5/istio-proxy/0.log"
Oct 29 22:17:08.898062 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:08.898034    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-trj5g_3973cd94-3216-4920-a993-0ea710831e20/payload-pre-processing/0.log"
Oct 29 22:17:09.088794 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:09.088765    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-d4fgq_3472b532-36f0-4a0d-9d04-27ec9a9f32e6/payload-processing/0.log"
Oct 29 22:17:09.271604 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:09.271584    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6d4df84b7f-nm9mn_a2536550-3bec-4756-b872-7b14a8eb78c3/router/0.log"
Oct 29 22:17:09.718599 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:17:09.718566    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:17:10.254428 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:10.254398    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-fqldg_70e1bafe-4b93-44d7-a179-44d5a82fd069/main/0.log"
Oct 29 22:17:10.448472 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:10.448449    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-xnbks_a85f7d35-4bff-45d0-a392-2ecf056c2051/main/0.log"
Oct 29 22:17:10.831697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:10.831668    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8fxd9d_696bd863-08fa-4694-843f-ab65f3ccfeed/main/0.log"
Oct 29 22:17:21.950078 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950043    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xbhcf/must-gather-x5snm"]
Oct 29 22:17:21.950501 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950408    4962 cpu_manager.go:414] "containerMap: removing container" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main"
Oct 29 22:17:21.950501 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950420    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main"
Oct 29 22:17:21.950501 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950431    4962 cpu_manager.go:414] "containerMap: removing container" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main"
Oct 29 22:17:21.950501 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950436    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main"
Oct 29 22:17:21.950501 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950446    4962 cpu_manager.go:414] "containerMap: removing container" podUID="132ed775-c9e6-41f1-b224-fff550c6e073" containerName="istio-proxy"
Oct 29 22:17:21.950501 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950452    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="132ed775-c9e6-41f1-b224-fff550c6e073" containerName="istio-proxy"
Oct 29 22:17:21.950501 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950464    4962 cpu_manager.go:414] "containerMap: removing container" podUID="adbf3c9f-a2e2-49b9-bef2-a9a0847b0374" containerName="revoke-keys"
Oct 29 22:17:21.950501 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950469    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adbf3c9f-a2e2-49b9-bef2-a9a0847b0374" containerName="revoke-keys"
Oct 29 22:17:21.950501 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950483    4962 cpu_manager.go:414] "containerMap: removing container" podUID="cee4891f-98dd-4ea8-a470-0f85ea62b849" containerName="payload-pre-processing"
Oct 29 22:17:21.950501 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950488    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee4891f-98dd-4ea8-a470-0f85ea62b849" containerName="payload-pre-processing"
Oct 29 22:17:21.950501 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950501    4962 cpu_manager.go:414] "containerMap: removing container" podUID="d37a81f7-dff2-429d-8dd1-c5d626299df8" containerName="payload-processing"
Oct 29 22:17:21.950501 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950506    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d37a81f7-dff2-429d-8dd1-c5d626299df8" containerName="payload-processing"
Oct 29 22:17:21.950939 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950513    4962 cpu_manager.go:414] "containerMap: removing container" podUID="d91917d2-1f66-4ee4-a312-57398651739e" containerName="istio-proxy"
Oct 29 22:17:21.950939 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950518    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d91917d2-1f66-4ee4-a312-57398651739e" containerName="istio-proxy"
Oct 29 22:17:21.950939 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950524    4962 cpu_manager.go:414] "containerMap: removing container" podUID="730edf47-8b07-477a-acfc-47faaaacfc84" containerName="istio-proxy"
Oct 29 22:17:21.950939 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950529    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730edf47-8b07-477a-acfc-47faaaacfc84" containerName="istio-proxy"
Oct 29 22:17:21.950939 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950547    4962 cpu_manager.go:414] "containerMap: removing container" podUID="8054bc19-6db7-462f-8af0-335903ce62f3" containerName="istio-proxy"
Oct 29 22:17:21.950939 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950552    4962 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8054bc19-6db7-462f-8af0-335903ce62f3" containerName="istio-proxy"
Oct 29 22:17:21.950939 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950605    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="8054bc19-6db7-462f-8af0-335903ce62f3" containerName="istio-proxy"
Oct 29 22:17:21.950939 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950614    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="e609098d-a77f-45bd-b87b-7e579dd0ed73" containerName="main"
Oct 29 22:17:21.950939 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950620    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9303c60-e8b1-4f7c-841f-811099ba8660" containerName="main"
Oct 29 22:17:21.950939 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950627    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="132ed775-c9e6-41f1-b224-fff550c6e073" containerName="istio-proxy"
Oct 29 22:17:21.950939 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950634    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="adbf3c9f-a2e2-49b9-bef2-a9a0847b0374" containerName="revoke-keys"
Oct 29 22:17:21.950939 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950642    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee4891f-98dd-4ea8-a470-0f85ea62b849" containerName="payload-pre-processing"
Oct 29 22:17:21.950939 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950648    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="d37a81f7-dff2-429d-8dd1-c5d626299df8" containerName="payload-processing"
Oct 29 22:17:21.950939 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950654    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="d91917d2-1f66-4ee4-a312-57398651739e" containerName="istio-proxy"
Oct 29 22:17:21.950939 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.950661    4962 memory_manager.go:365] "RemoveStaleState removing state" podUID="730edf47-8b07-477a-acfc-47faaaacfc84" containerName="istio-proxy"
Oct 29 22:17:21.953548 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.953530    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xbhcf/must-gather-x5snm"
Oct 29 22:17:21.980606 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.980579    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xbhcf\"/\"default-dockercfg-t86kg\""
Oct 29 22:17:21.980735 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.980636    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xbhcf\"/\"openshift-service-ca.crt\""
Oct 29 22:17:21.980735 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.980639    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xbhcf\"/\"kube-root-ca.crt\""
Oct 29 22:17:21.982462 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:21.982440    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xbhcf/must-gather-x5snm"]
Oct 29 22:17:22.045019 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.044999    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-d9gbf/must-gather-zf7gt"]
Oct 29 22:17:22.054881 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.054854    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bzkjc/must-gather-7zq99"]
Oct 29 22:17:22.054997 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.054977    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d9gbf/must-gather-zf7gt"
Oct 29 22:17:22.058904 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.058881    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bzkjc/must-gather-7zq99"
Oct 29 22:17:22.069790 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.069751    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-d9gbf\"/\"default-dockercfg-q8lh9\""
Oct 29 22:17:22.070567 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.070545    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d9gbf\"/\"openshift-service-ca.crt\""
Oct 29 22:17:22.070709 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.070641    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d9gbf\"/\"kube-root-ca.crt\""
Oct 29 22:17:22.071699 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.071677    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d9gbf/must-gather-zf7gt"]
Oct 29 22:17:22.074302 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.074265    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bzkjc/must-gather-7zq99"]
Oct 29 22:17:22.124542 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.124521    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5m5b\" (UniqueName: \"kubernetes.io/projected/cf1c2246-a046-4546-99bc-eec415d74fb7-kube-api-access-q5m5b\") pod \"must-gather-x5snm\" (UID: \"cf1c2246-a046-4546-99bc-eec415d74fb7\") " pod="openshift-must-gather-xbhcf/must-gather-x5snm"
Oct 29 22:17:22.124669 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.124564    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cf1c2246-a046-4546-99bc-eec415d74fb7-must-gather-output\") pod \"must-gather-x5snm\" (UID: \"cf1c2246-a046-4546-99bc-eec415d74fb7\") " pod="openshift-must-gather-xbhcf/must-gather-x5snm"
Oct 29 22:17:22.135746 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.135722    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bzkjc\"/\"openshift-service-ca.crt\""
Oct 29 22:17:22.135746 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.135730    4962 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bzkjc\"/\"kube-root-ca.crt\""
Oct 29 22:17:22.169586 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.169566    4962 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bzkjc\"/\"default-dockercfg-78mjz\""
Oct 29 22:17:22.225329 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.225255    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sglm\" (UniqueName: \"kubernetes.io/projected/c341dc70-d0d6-469f-93a5-ab62134bf271-kube-api-access-2sglm\") pod \"must-gather-7zq99\" (UID: \"c341dc70-d0d6-469f-93a5-ab62134bf271\") " pod="openshift-must-gather-bzkjc/must-gather-7zq99"
Oct 29 22:17:22.225422 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.225387    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbfn9\" (UniqueName: \"kubernetes.io/projected/1dc24071-1c12-4722-a30d-3386a4aa0947-kube-api-access-fbfn9\") pod \"must-gather-zf7gt\" (UID: \"1dc24071-1c12-4722-a30d-3386a4aa0947\") " pod="openshift-must-gather-d9gbf/must-gather-zf7gt"
Oct 29 22:17:22.225422 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.225417    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5m5b\" (UniqueName: \"kubernetes.io/projected/cf1c2246-a046-4546-99bc-eec415d74fb7-kube-api-access-q5m5b\") pod \"must-gather-x5snm\" (UID: \"cf1c2246-a046-4546-99bc-eec415d74fb7\") " pod="openshift-must-gather-xbhcf/must-gather-x5snm"
Oct 29 22:17:22.225549 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.225448    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1dc24071-1c12-4722-a30d-3386a4aa0947-must-gather-output\") pod \"must-gather-zf7gt\" (UID: \"1dc24071-1c12-4722-a30d-3386a4aa0947\") " pod="openshift-must-gather-d9gbf/must-gather-zf7gt"
Oct 29 22:17:22.225715 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.225692    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cf1c2246-a046-4546-99bc-eec415d74fb7-must-gather-output\") pod \"must-gather-x5snm\" (UID: \"cf1c2246-a046-4546-99bc-eec415d74fb7\") " pod="openshift-must-gather-xbhcf/must-gather-x5snm"
Oct 29 22:17:22.225781 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.225735    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c341dc70-d0d6-469f-93a5-ab62134bf271-must-gather-output\") pod \"must-gather-7zq99\" (UID: \"c341dc70-d0d6-469f-93a5-ab62134bf271\") " pod="openshift-must-gather-bzkjc/must-gather-7zq99"
Oct 29 22:17:22.225994 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.225972    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cf1c2246-a046-4546-99bc-eec415d74fb7-must-gather-output\") pod \"must-gather-x5snm\" (UID: \"cf1c2246-a046-4546-99bc-eec415d74fb7\") " pod="openshift-must-gather-xbhcf/must-gather-x5snm"
Oct 29 22:17:22.247725 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.247702    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5m5b\" (UniqueName: \"kubernetes.io/projected/cf1c2246-a046-4546-99bc-eec415d74fb7-kube-api-access-q5m5b\") pod \"must-gather-x5snm\" (UID: \"cf1c2246-a046-4546-99bc-eec415d74fb7\") " pod="openshift-must-gather-xbhcf/must-gather-x5snm"
Oct 29 22:17:22.268149 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.268129    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xbhcf/must-gather-x5snm"
Oct 29 22:17:22.328786 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.327306    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sglm\" (UniqueName: \"kubernetes.io/projected/c341dc70-d0d6-469f-93a5-ab62134bf271-kube-api-access-2sglm\") pod \"must-gather-7zq99\" (UID: \"c341dc70-d0d6-469f-93a5-ab62134bf271\") " pod="openshift-must-gather-bzkjc/must-gather-7zq99"
Oct 29 22:17:22.328786 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.327431    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbfn9\" (UniqueName: \"kubernetes.io/projected/1dc24071-1c12-4722-a30d-3386a4aa0947-kube-api-access-fbfn9\") pod \"must-gather-zf7gt\" (UID: \"1dc24071-1c12-4722-a30d-3386a4aa0947\") " pod="openshift-must-gather-d9gbf/must-gather-zf7gt"
Oct 29 22:17:22.328786 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.327479    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1dc24071-1c12-4722-a30d-3386a4aa0947-must-gather-output\") pod \"must-gather-zf7gt\" (UID: \"1dc24071-1c12-4722-a30d-3386a4aa0947\") " pod="openshift-must-gather-d9gbf/must-gather-zf7gt"
Oct 29 22:17:22.328786 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.327550    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c341dc70-d0d6-469f-93a5-ab62134bf271-must-gather-output\") pod \"must-gather-7zq99\" (UID: \"c341dc70-d0d6-469f-93a5-ab62134bf271\") " pod="openshift-must-gather-bzkjc/must-gather-7zq99"
Oct 29 22:17:22.328786 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.327977    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c341dc70-d0d6-469f-93a5-ab62134bf271-must-gather-output\") pod \"must-gather-7zq99\" (UID: \"c341dc70-d0d6-469f-93a5-ab62134bf271\") " pod="openshift-must-gather-bzkjc/must-gather-7zq99"
Oct 29 22:17:22.328786 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.328004    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1dc24071-1c12-4722-a30d-3386a4aa0947-must-gather-output\") pod \"must-gather-zf7gt\" (UID: \"1dc24071-1c12-4722-a30d-3386a4aa0947\") " pod="openshift-must-gather-d9gbf/must-gather-zf7gt"
Oct 29 22:17:22.349590 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.349531    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbfn9\" (UniqueName: \"kubernetes.io/projected/1dc24071-1c12-4722-a30d-3386a4aa0947-kube-api-access-fbfn9\") pod \"must-gather-zf7gt\" (UID: \"1dc24071-1c12-4722-a30d-3386a4aa0947\") " pod="openshift-must-gather-d9gbf/must-gather-zf7gt"
Oct 29 22:17:22.349706 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.349655    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sglm\" (UniqueName: \"kubernetes.io/projected/c341dc70-d0d6-469f-93a5-ab62134bf271-kube-api-access-2sglm\") pod \"must-gather-7zq99\" (UID: \"c341dc70-d0d6-469f-93a5-ab62134bf271\") " pod="openshift-must-gather-bzkjc/must-gather-7zq99"
Oct 29 22:17:22.369523 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.369503    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d9gbf/must-gather-zf7gt"
Oct 29 22:17:22.376530 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.376509    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bzkjc/must-gather-7zq99"
Oct 29 22:17:22.571364 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.571341    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xbhcf/must-gather-x5snm"]
Oct 29 22:17:22.574087 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:17:22.574064    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf1c2246_a046_4546_99bc_eec415d74fb7.slice/crio-c24711598f27f60674cf00b19eb5dc6c8648aabc75cf18fcce95a8de1f11914d WatchSource:0}: Error finding container c24711598f27f60674cf00b19eb5dc6c8648aabc75cf18fcce95a8de1f11914d: Status 404 returned error can't find the container with id c24711598f27f60674cf00b19eb5dc6c8648aabc75cf18fcce95a8de1f11914d
Oct 29 22:17:22.575748 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.575732    4962 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:17:22.659382 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.659353    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xbhcf/must-gather-x5snm" event={"ID":"cf1c2246-a046-4546-99bc-eec415d74fb7","Type":"ContainerStarted","Data":"c24711598f27f60674cf00b19eb5dc6c8648aabc75cf18fcce95a8de1f11914d"}
Oct 29 22:17:22.670195 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.670143    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d9gbf/must-gather-zf7gt"]
Oct 29 22:17:22.672245 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:17:22.672219    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1dc24071_1c12_4722_a30d_3386a4aa0947.slice/crio-1a3442475893cba26a80b4b7a22c49304fd867026d1f22fbc440d5244e5438d6 WatchSource:0}: Error finding container 1a3442475893cba26a80b4b7a22c49304fd867026d1f22fbc440d5244e5438d6: Status 404 returned error can't find the container with id 1a3442475893cba26a80b4b7a22c49304fd867026d1f22fbc440d5244e5438d6
Oct 29 22:17:22.678510 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:22.678486    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bzkjc/must-gather-7zq99"]
Oct 29 22:17:22.680171 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:17:22.680149    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc341dc70_d0d6_469f_93a5_ab62134bf271.slice/crio-45992022254c7c7bcbef2dc59aa27ea5b9c1da38e652375e1162ed0173b5fdd7 WatchSource:0}: Error finding container 45992022254c7c7bcbef2dc59aa27ea5b9c1da38e652375e1162ed0173b5fdd7: Status 404 returned error can't find the container with id 45992022254c7c7bcbef2dc59aa27ea5b9c1da38e652375e1162ed0173b5fdd7
Oct 29 22:17:23.702307 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:23.696809    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bzkjc/must-gather-7zq99" event={"ID":"c341dc70-d0d6-469f-93a5-ab62134bf271","Type":"ContainerStarted","Data":"bc49dc09c0f1767f97c4931ea85e79491fbcdec8e842e1c42ca5ac84d108fb9b"}
Oct 29 22:17:23.702307 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:23.696875    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bzkjc/must-gather-7zq99" event={"ID":"c341dc70-d0d6-469f-93a5-ab62134bf271","Type":"ContainerStarted","Data":"45992022254c7c7bcbef2dc59aa27ea5b9c1da38e652375e1162ed0173b5fdd7"}
Oct 29 22:17:23.719933 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:23.713649    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d9gbf/must-gather-zf7gt" event={"ID":"1dc24071-1c12-4722-a30d-3386a4aa0947","Type":"ContainerStarted","Data":"1a3442475893cba26a80b4b7a22c49304fd867026d1f22fbc440d5244e5438d6"}
Oct 29 22:17:24.742309 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:24.737873    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bzkjc/must-gather-7zq99" event={"ID":"c341dc70-d0d6-469f-93a5-ab62134bf271","Type":"ContainerStarted","Data":"29071b3aa241beeec9a16d82c770ba52e6f545eb1e1a3e835de64e65f5ab3999"}
Oct 29 22:17:24.778465 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:24.777827    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bzkjc/must-gather-7zq99" podStartSLOduration=2.107202909 podStartE2EDuration="2.777810878s" totalImagesPullingTime="670.607969ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:17:24.776311235 +0000 UTC m=+1865.849268199" watchObservedRunningTime="2026-10-29 22:17:24.777810878 +0000 UTC m=+1865.850767843"
Oct 29 22:17:27.265560 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:27.265195    4962 scope.go:122] "RemoveContainer" containerID="678eac811730ab90c261ea74bfe16e0c5ed7e0a918d54786252bd39a6f922e73"
Oct 29 22:17:27.604063 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:27.603988    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hwq7z_676904dc-4d4f-4bd6-b380-357d5d9a29d4/global-pull-secret-syncer/0.log"
Oct 29 22:17:27.970457 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:27.970426    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-4w7w5_d172a43b-a20c-4fc3-a1cc-77a697312061/connection-checker/0.log"
Oct 29 22:17:28.091517 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:28.091484    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-fkzj6_1065c119-4756-47ad-960c-c99ee1fae963/connection-checker/0.log"
Oct 29 22:17:28.245622 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:28.245537    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-lqjq8_73358782-c31a-4cb4-a0c2-377a1bceae4e/connection-checker/0.log"
Oct 29 22:17:28.518935 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:28.518836    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-g2rdv_af572bfb-940b-4226-b91f-d06554b40d3c/konnectivity-agent/0.log"
Oct 29 22:17:28.920304 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:28.920186    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-146.ec2.internal_4db57e6b6465b77567369446b9e61568/haproxy/0.log"
Oct 29 22:17:35.996317 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:35.988563    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-7bbc6bb9d7-92hd8_ac207213-067e-4673-b4b2-630469c878a8/kuadrant-console-plugin/0.log"
Oct 29 22:17:38.041602 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:38.041573    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-68669df9c4-ktqlt_3391c781-0e29-49eb-a190-20229a0ab066/manager/0.log"
Oct 29 22:17:38.939002 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:38.931627    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d9gbf/must-gather-zf7gt" event={"ID":"1dc24071-1c12-4722-a30d-3386a4aa0947","Type":"ContainerStarted","Data":"76559c8d0f39fd0a6b317be0d6b78d688584c63b801cdc19827ae986754b56fb"}
Oct 29 22:17:38.939002 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:38.931672    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d9gbf/must-gather-zf7gt" event={"ID":"1dc24071-1c12-4722-a30d-3386a4aa0947","Type":"ContainerStarted","Data":"fd5df65dcc2dd89a6c82c0013069c6b1e8e4da93044454094f7dc964c1246312"}
Oct 29 22:17:38.939002 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:38.937587    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xbhcf/must-gather-x5snm" event={"ID":"cf1c2246-a046-4546-99bc-eec415d74fb7","Type":"ContainerStarted","Data":"c80e7219e60eee48bd6982579ea73c8c67f5f2842e83186da20a27fa5fee5ea6"}
Oct 29 22:17:38.939002 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:38.937629    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xbhcf/must-gather-x5snm" event={"ID":"cf1c2246-a046-4546-99bc-eec415d74fb7","Type":"ContainerStarted","Data":"46a3b1cd3550d0ef7ae29502ba3d2078edab9c66cfe10803f07900b715e4b5e6"}
Oct 29 22:17:38.976231 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:38.976169    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-d9gbf/must-gather-zf7gt" podStartSLOduration=1.8526812179999999 podStartE2EDuration="16.976146061s" totalImagesPullingTime="15.123464843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:17:38.972658313 +0000 UTC m=+1880.045615277" watchObservedRunningTime="2026-10-29 22:17:38.976146061 +0000 UTC m=+1880.049103024"
Oct 29 22:17:39.004587 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.004544    4962 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt"]
Oct 29 22:17:39.017011 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.016955    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-xbhcf/must-gather-x5snm" podStartSLOduration=2.7735421970000003 podStartE2EDuration="18.016936169s" totalImagesPullingTime="15.243393972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:17:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:17:39.016055902 +0000 UTC m=+1880.089012867" watchObservedRunningTime="2026-10-29 22:17:39.016936169 +0000 UTC m=+1880.089893134"
Oct 29 22:17:39.106677 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.106641    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt"
Oct 29 22:17:39.117902 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.117681    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt"]
Oct 29 22:17:39.160028 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.159997    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8310b07d-6061-4816-9115-79a0bd413f9f-sys\") pod \"perf-node-gather-daemonset-2q2pt\" (UID: \"8310b07d-6061-4816-9115-79a0bd413f9f\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt"
Oct 29 22:17:39.160028 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.160031    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8310b07d-6061-4816-9115-79a0bd413f9f-podres\") pod \"perf-node-gather-daemonset-2q2pt\" (UID: \"8310b07d-6061-4816-9115-79a0bd413f9f\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt"
Oct 29 22:17:39.160326 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.160059    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flgng\" (UniqueName: \"kubernetes.io/projected/8310b07d-6061-4816-9115-79a0bd413f9f-kube-api-access-flgng\") pod \"perf-node-gather-daemonset-2q2pt\" (UID: \"8310b07d-6061-4816-9115-79a0bd413f9f\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt"
Oct 29 22:17:39.160326 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.160205    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8310b07d-6061-4816-9115-79a0bd413f9f-lib-modules\") pod \"perf-node-gather-daemonset-2q2pt\" (UID: \"8310b07d-6061-4816-9115-79a0bd413f9f\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt"
Oct 29 22:17:39.160518 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.160318    4962 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8310b07d-6061-4816-9115-79a0bd413f9f-proc\") pod \"perf-node-gather-daemonset-2q2pt\" (UID: \"8310b07d-6061-4816-9115-79a0bd413f9f\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt"
Oct 29 22:17:39.261817 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.261355    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8310b07d-6061-4816-9115-79a0bd413f9f-lib-modules\") pod \"perf-node-gather-daemonset-2q2pt\" (UID: \"8310b07d-6061-4816-9115-79a0bd413f9f\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt"
Oct 29 22:17:39.261817 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.261422    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8310b07d-6061-4816-9115-79a0bd413f9f-proc\") pod \"perf-node-gather-daemonset-2q2pt\" (UID: \"8310b07d-6061-4816-9115-79a0bd413f9f\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt"
Oct 29 22:17:39.261817 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.261531    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8310b07d-6061-4816-9115-79a0bd413f9f-sys\") pod \"perf-node-gather-daemonset-2q2pt\" (UID: \"8310b07d-6061-4816-9115-79a0bd413f9f\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt"
Oct 29 22:17:39.261817 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.261559    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8310b07d-6061-4816-9115-79a0bd413f9f-podres\") pod \"perf-node-gather-daemonset-2q2pt\" (UID: \"8310b07d-6061-4816-9115-79a0bd413f9f\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt"
Oct 29 22:17:39.261817 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.261605    4962 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flgng\" (UniqueName: \"kubernetes.io/projected/8310b07d-6061-4816-9115-79a0bd413f9f-kube-api-access-flgng\") pod \"perf-node-gather-daemonset-2q2pt\" (UID: \"8310b07d-6061-4816-9115-79a0bd413f9f\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt"
Oct 29 22:17:39.262257 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.262030    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8310b07d-6061-4816-9115-79a0bd413f9f-lib-modules\") pod \"perf-node-gather-daemonset-2q2pt\" (UID: \"8310b07d-6061-4816-9115-79a0bd413f9f\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt"
Oct 29 22:17:39.262257 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.262094    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8310b07d-6061-4816-9115-79a0bd413f9f-proc\") pod \"perf-node-gather-daemonset-2q2pt\" (UID: \"8310b07d-6061-4816-9115-79a0bd413f9f\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt"
Oct 29 22:17:39.262257 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.262139    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8310b07d-6061-4816-9115-79a0bd413f9f-sys\") pod \"perf-node-gather-daemonset-2q2pt\" (UID: \"8310b07d-6061-4816-9115-79a0bd413f9f\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt"
Oct 29 22:17:39.262257 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.262196    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8310b07d-6061-4816-9115-79a0bd413f9f-podres\") pod \"perf-node-gather-daemonset-2q2pt\" (UID: \"8310b07d-6061-4816-9115-79a0bd413f9f\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt"
Oct 29 22:17:39.298717 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.298680    4962 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flgng\" (UniqueName: \"kubernetes.io/projected/8310b07d-6061-4816-9115-79a0bd413f9f-kube-api-access-flgng\") pod \"perf-node-gather-daemonset-2q2pt\" (UID: \"8310b07d-6061-4816-9115-79a0bd413f9f\") " pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt"
Oct 29 22:17:39.470132 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.470099    4962 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt"
Oct 29 22:17:39.729523 ip-10-0-129-146 kubenswrapper[4962]: E1029 22:17:39.729487    4962 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dtgwl" containerName="extractor"
Oct 29 22:17:39.788932 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.788471    4962 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt"]
Oct 29 22:17:39.792764 ip-10-0-129-146 kubenswrapper[4962]: W1029 22:17:39.792733    4962 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8310b07d_6061_4816_9115_79a0bd413f9f.slice/crio-137629258f90ca833925c4787870eb526fc97e3f17a59eadd07f7a45c9c05a53 WatchSource:0}: Error finding container 137629258f90ca833925c4787870eb526fc97e3f17a59eadd07f7a45c9c05a53: Status 404 returned error can't find the container with id 137629258f90ca833925c4787870eb526fc97e3f17a59eadd07f7a45c9c05a53
Oct 29 22:17:39.945769 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.945737    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt" event={"ID":"8310b07d-6061-4816-9115-79a0bd413f9f","Type":"ContainerStarted","Data":"6f6e86059a809e7f5f218085398d45b6f68fad1ca505d0c7535429905219f3ea"}
Oct 29 22:17:39.945934 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.945786    4962 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt" event={"ID":"8310b07d-6061-4816-9115-79a0bd413f9f","Type":"ContainerStarted","Data":"137629258f90ca833925c4787870eb526fc97e3f17a59eadd07f7a45c9c05a53"}
Oct 29 22:17:39.946218 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.946200    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt"
Oct 29 22:17:39.990420 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:39.990360    4962 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt" podStartSLOduration=1.9903415660000001 podStartE2EDuration="1.990341566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:17:39.988132496 +0000 UTC m=+1881.061089461" watchObservedRunningTime="2026-10-29 22:17:39.990341566 +0000 UTC m=+1881.063298531"
Oct 29 22:17:40.369392 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:40.369362    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-767df45bdd-6djqs_68e70e0c-c896-4641-b5d7-f97e8bf2ce79/manager/0.log"
Oct 29 22:17:40.505959 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:40.505931    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-767df45bdd-6djqs_68e70e0c-c896-4641-b5d7-f97e8bf2ce79/manager/0.log"
Oct 29 22:17:40.650205 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:40.650111    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-6p66l_1eb2f410-512d-4e3e-838a-87c1e3b47207/cluster-monitoring-operator/0.log"
Oct 29 22:17:41.757402 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:41.757281    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tc8tb_d526d422-6b0e-46ab-a904-a4c49ae5de79/node-exporter/0.log"
Oct 29 22:17:41.879440 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:41.879401    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tc8tb_d526d422-6b0e-46ab-a904-a4c49ae5de79/kube-rbac-proxy/0.log"
Oct 29 22:17:42.005102 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:42.005076    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tc8tb_d526d422-6b0e-46ab-a904-a4c49ae5de79/init-textfile/0.log"
Oct 29 22:17:43.296414 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:43.296380    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-767df45bdd-6djqs_68e70e0c-c896-4641-b5d7-f97e8bf2ce79/manager/0.log"
Oct 29 22:17:43.474215 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:43.474186    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-767df45bdd-6djqs_68e70e0c-c896-4641-b5d7-f97e8bf2ce79/manager/0.log"
Oct 29 22:17:46.281182 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:46.281151    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-767df45bdd-6djqs_68e70e0c-c896-4641-b5d7-f97e8bf2ce79/manager/0.log"
Oct 29 22:17:46.424744 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:46.424711    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-767df45bdd-6djqs_68e70e0c-c896-4641-b5d7-f97e8bf2ce79/manager/0.log"
Oct 29 22:17:46.705885 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:46.705849    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-9np86_72352328-6f28-47ae-9190-0176cedaed4a/networking-console-plugin/0.log"
Oct 29 22:17:46.975756 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:46.975676    4962 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bzkjc/perf-node-gather-daemonset-2q2pt"
Oct 29 22:17:48.265739 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:48.265694    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/2.log"
Oct 29 22:17:48.270020 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:48.269989    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-jdps8_f7ed52a4-5ca3-4c5c-a2b5-f4bf6fb28600/console-operator/3.log"
Oct 29 22:17:51.200131 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:51.200103    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-cv94k_530bea30-e132-4b1a-9c6d-e421c216060a/volume-data-source-validator/0.log"
Oct 29 22:17:54.379697 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:54.379662    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-swlfh_99c9881d-05b3-4953-b484-3ce51520b84a/dns/0.log"
Oct 29 22:17:54.497110 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:54.497075    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-swlfh_99c9881d-05b3-4953-b484-3ce51520b84a/kube-rbac-proxy/0.log"
Oct 29 22:17:55.181827 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:55.181794    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rrmxb_fc992647-89dd-44cc-ba19-3fdb03e125de/dns-node-resolver/0.log"
Oct 29 22:17:56.239038 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:56.238992    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-bbmrg_2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5/discovery/0.log"
Oct 29 22:17:56.506947 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:56.506871    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-bbmrg_2b5c68ae-59b4-43ab-a8a6-cbfb407d51f5/discovery/0.log"
Oct 29 22:17:56.763592 ip-10-0-129-146 kubenswrapper[4962]: I1029 22:17:56.763499    4962 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-knxtf_8428734b-4a9f-4df9-ac06-81074198a9ba/node-ca/0.log"